You downloaded this. Here is what it is.

First, the unit you are holding

One file is a slice, not a dataset

A dataset is the record. A granule is one file out of it. Almost every confusing thing about Earthdata follows from that distinction.

NASA does not hand you a dataset. It hands you granules — the smallest slice it is willing to distribute. Download a year of this product and you have 365 files that are individually meaningless and collectively a scientific record.

This is the first thing that trips people up. You go looking for data and you get a file with a name like the one above, which is one slice out of a record running for years. The collection identifier names the whole record; the granule is one slice of it.

Second, the container

What kind of file is this, really?

HDF5, netCDF-4, HDF4 and GeoTIFF are containers, not data. Which one you have decides which software will open it.

The file extension is a label someone typed. It is not enforced by anything, and renaming a file does not change what is inside it. So the honest first question is: what container is this actually written in, and what will open it?

Third, and most important

Nobody measured most of these cells

Level 3 means gridded. Gridded means an algorithm filled the space between what the satellites actually saw.

Here is the idea that changes how you read every number in this file.

Nobody measured the quantity in those 6,480,000 cells. Satellites do not see a grid. A satellite sees a swath — a strip of ground traced out underneath it as it orbits. Between one pass and the next, that patch of ground is not being watched at all. A grid covering the whole planet with no holes in it is something an algorithm produced from those strips.

NASA labels how far a product has travelled from the raw signal. The number appears in the filename of almost every Earthdata product, and it is the fastest way to know what you are dealing with.

A cell reading 0.4 is the algorithm’s best estimate for that cell, not a reading from an instrument that looked at it. Some cells were observed directly seconds before the window closed. Others were last seen an hour ago and have been carried forward on modelled winds. The next chapter is how to tell which.

Fourth, how it was made

Your file shows its working

Every array in here is the output of a step. Knowing the steps is how you know what the numbers mean.

IMERG stands for Integrated Multi-satellitE Retrievals for GPM. The name is the method: it takes what a constellation of satellites saw, and stitches it into one continuous rainfall field covering the whole planet, with no gaps.

This is the part worth slowing down for, because the steps below are not background reading — several of them left their output in your file, in the arrays named beside them. Most people take the headline variable and never look. But if you want to know why a value is what it is, the working is right there.

1. Passive microwave

   Radiometers on roughly a dozen satellites see through cloud and estimate rain rate directly. Best estimate available, but each satellite only passes over a given spot every few hours.

   visible in your file asMWprecipitation

2. Morphing

   Between passes, the last observation is pushed along using winds from a weather reanalysis, so a storm keeps moving on the grid even when nothing is watching it. The Early run can only push forward in time; Late and Final also interpolate backward from the next pass, which is more accurate.

   changes the other fields in place; leaves no variable behind

3. Infrared fill

   Geostationary satellites watch continuously but only see cloud-top temperature, a weak proxy for rain. Used where microwave and morphing leave a gap.

   this stage’s output is not kept in this granule

4. Merge

   The sources are combined into one field, weighted by how trustworthy each is at that place and time. This is the finished product for the Early and Late runs.

   this stage’s output is not kept in this granule

5. Gauge calibration not in the Late run

   Final run only. Once a month of ground rain-gauge records has been collected and quality-controlled, the merged field is adjusted to agree with it. This is why Final takes months to appear, and why it is the one to use for anything you publish.

   visible in your file asprecipitation

Fifth, the label on the outside

The filename is a fingerprint

Every character is doing a job. Nothing in the name is arbitrary and nothing repeats by accident.

An Earthdata filename is not decoration. It is a structured record, and it is the only part of the file you can read without opening anything. Once you can decode it you can sort, filter and sanity-check a directory of ten thousand granules from the shell.

The catch: the key to reading it is not inside the file. It lives in a specification document on a different server, nowhere linked from the download page. That is the gap this whole exercise exists to close.

Sixth, the stamp inside

The extension can lie; the opening bytes cannot

Renaming a file changes its label and nothing else. Formats identify themselves from the inside.

Every file format stamps its identity into its opening bytes. The stamp is written by whatever program created the file, and unlike the extension, it does not survive being renamed. Reading it is how a program knows what it is holding before it commits to a parser.

Bytes two through four spell HDF in plain ASCII, so a damaged file often still announces what it was trying to be. The bytes either side are deliberate traps: they are chosen to get mangled if the file crosses a connection that “helpfully” converts line endings, which turns a silent corruption into an obvious one.

Seventh, what else you need

One file is rarely the whole answer

Products come in families. The one you downloaded may be a piece of a set, and useless on its own.

Related products, before you write a loop. NASA splits products along lines that make sense to the production system and are invisible from the download page. Knowing which sibling you actually want saves more time than any amount of clever code.

One boundary to respect: the V07 Final record stops in September 2025 while the upstream products move to V08, and a full V08 reprocessing of the record follows. Treat a version change exactly like a collection change — never mix versions in one analysis.

Opening it up

Inside is a folder tree

Groups, arrays and notes. Once you can see the shape of it, the file stops being opaque.

Open the container and you find something laid out like the folders on your computer. A group is a folder. Inside it are arrays. Attached to each array are attributes — short labelled notes saying what the array is and what to watch out for.

• data The actual values. This is what you came for.
• axis Coordinate values. Row 400, column 900 of a data array means nothing until you look up what position those indices point at.
• bounds The edges of each cell rather than its centre. You need edges to compute an area correctly.

The notes on the arrays

The sticky notes are the documentation

Units, descriptions, missing-data markers. This is the metadata everyone tells you to read, and here is what is in it.

The notes attached to an array are where a self-describing file earns the name. They are also where the traps are, because a note that is missing does not raise an error — software just carries on without it.

The one that catches everybody

−9999.9 is not a rainfall measurement

Every product has a way of being silently wrong. This is this product’s.

The algorithm cannot always produce an estimate. Those cells still need a number written into them, because the array is a rectangle and rectangles do not have holes. A fill value goes in: chosen to be so far outside the plausible range that nobody could mistake it for rain. It works right up until somebody calls .mean().

fill = ds[path].attrs['_FillValue']
a = ds[path][:]
a = np.where(np.isclose(a, fill, rtol=0, atol=0.01), np.nan, a)
mean = np.nanmean(a)

Onto a real map

Three maps walk in; one is right

A regular grid loads on the first try, which is the problem: wrong axis order and wrong row direction also load on the first try.

Sooner or later this file meets a map program. A gridded product is the easy case — a regular lattice is exactly what raster software expects — and that is precisely what makes it dangerous. The load succeeds either way. Whether the map is right depends on two conventions the software has to guess: which axis is which, and which way the rows run.

Two specifics for this product. First, the grid is stored in (time, lon, lat) order — longitude varies along the first spatial axis, backwards from what raster tools expect, so a naive load renders sideways until you transpose. Second, common netCDF readers assume rows run bottom-up, which flips the world vertically. Neither raises an error; both produce a confident, wrong map. If your destination is QGIS or ArcGIS, skip the fight: PPS publishes a GIS-ready GeoTIFF version of every IMERG granule.

The tell is always geography. Before trusting any load, find one coastline you know and check it — a sideways or mirrored continent takes one second to spot and one line of code to fix, but only if you look.

When

The clock does not start in 1970

Time is a count of seconds since a reference date, and the reference is rarely the one your language assumes.

Time in a science data file is almost never a date. It is a count of seconds since some reference moment, and the reference is written in the array’s units string. Read the wrong reference and every timestamp you produce is confidently, consistently wrong. The one everyone assumes is 1 January 1970, the Unix epoch. Earth science products frequently do not use it.

Two more things to internalise while you are here. Everything is UTC, never local — a granule labelled 20:48 covers the evening in Lagos and the afternoon in Denver. And every product picks a convention for its end stamp: the last instant inside the window, or the first instant of the next one. Get it wrong and you double-count or drop a slice everywhere two granules meet. Where this guide knows the product’s convention it has already applied it — the times and durations shown above describe the true window, and the filename chapter explains the raw stamp.

House rules

Conventions, and which ones matter to you

Conformance checking sounds like paperwork. Some of it is. Some of it is the difference between a right and a wrong answer.

Now that you know what an attribute is, this will make sense. NASA publishes house rules — the DIWG recommendations — about how a data file should be built, so that software written by someone who has never seen your product can still read it. Alongside those sit the CF conventions, which fix the exact spelling of common attributes.

Files break these rules all the time, including files produced by NASA. That is not a scandal. What matters is which rule, because the consequences run from “a catalogue lists this slightly wrong” to “your average is off by three orders of magnitude.” Sorted that way, not by rule number:

One habit worth taking away: when a tool says a file is non-conformant, the useful question is never is this bad. It is what would I have got wrong if nobody had told me. Most of the time the answer is nothing. Occasionally it is everything.

More than one file

The same product, packaged two ways

Checking a folder answers a question a single file cannot: is what you found true of this granule, or of the whole product?

One granule tells you about one granule. Run the same checks over a folder and a different question opens: does the answer change from file to file?

Mostly it should not. How a product is packaged is decided once, by the algorithm team, and applies to everything they publish. A row where every file agrees is telling you about the product. A row where they disagree is telling you that two different things are sitting in your folder.

Dwell on any row marked differs. It usually means mixed cadences or mixed containers — half-hourly HDF5 alongside daily netCDF-4, say. Same algorithm, same version, but written by different code, so they store time differently and spell their attributes differently. A loop that reads one correctly can quietly misread the other.