What is a shapefile and which files does it need?

Where the shapefile comes from

Esri introduced the shapefile in the early 1990s with its desktop program ArcView, as a simple way to store points, lines and polygons together with a table of attributes. In 1998 Esri published the ESRI Shapefile Technical Description, a white paper that documents the layout of the geometry and index files. Because the description was public and the format was simple, other software could read and write it without a licence, and the shapefile became the common exchange format of desktop GIS.

That is still its main strength today. Government open data portals, surveyors, planning offices and utilities publish shapefiles because almost every GIS program, from QGIS and ArcGIS Pro to GDAL, PostGIS loaders and many CAD programs, can read them. The format also shows its age, and the limits described below are the reason newer formats exist.

The files that make up a shapefile

A shapefile is a set of files that share a base name, such as parcels.shp, parcels.shx and parcels.dbf. Three of them are mandatory. The .shp file stores the geometry of every feature as a list of coordinates. The .shx file is an index with the position of each record in the .shp, so that programs can jump straight to feature number 5,000 instead of reading everything before it. The .dbf file is the attribute table in the dBASE format, with one row per feature in the same order as the shapes. The link between a shape and its row is nothing more than that order.

The other files are optional. The .prj file holds the coordinate system as a line of well-known text (WKT), in the dialect Esri uses. The .cpg file is a tiny text file naming the character encoding of the .dbf, such as UTF-8 or 1252. The .sbn and .sbx files are a spatial index used by Esri software, and .qix is a different spatial index written by GDAL and MapServer; both only speed up drawing and searching in large files. The .shp.xml file carries metadata such as a description, the author and the date. You may also see .atx files for attribute indexes. If an optional file is missing, programs can still read the data, but without .prj the location is guesswork, and without .cpg special characters can go wrong.

The base names must match exactly, including upper and lower case on systems that care about it. Renaming only roads.shp to streets.shp breaks the set. Rename, copy and send all parts together, and the simplest way to keep them together is a .zip of the whole set.

One geometry type per shapefile

Each shapefile declares one shape type in its header, and every feature must be of that type or empty. The main types are Point, MultiPoint, PolyLine (lines) and Polygon, each in a plain 2D version, a version with Z values for height and a version with M values for measures, such as the distance along a route. A rarely used MultiPatch type describes 3D surfaces. A file cannot mix points with polygons, so a dataset of buildings and their entrances has to be stored as two shapefiles.

Polygons are stored as rings. In the published description, the outer boundary runs clockwise and holes run counter-clockwise, and a polygon with several separate parts, such as a country with islands, is a single feature with several rings. The format stores no topology: two neighbouring parcels each store their shared border separately, and nothing in the file guarantees that the lines match.

Limits of the .dbf attribute table

Many of the shapefile's quirks come from the dBASE table format, which dates from the 1980s. Field names can have at most 10 characters, so Population_2020 has to become something like POP_2020, and some programs also turn them into upper case. Text fields hold at most 254 characters. The number of fields is limited too; 255 is the figure usually given.

The data types are few. Text is stored as character fields of fixed width, numbers as numeric or floating point fields with a fixed width and number of decimals, true and false as logical fields, and dates as date fields of the form year, month, day. There is no time of day and no time zone, so a timestamp has to be stored as text or split into fields. There is no proper null value either: an empty number is usually written as blanks or as zero, depending on the program, and some readers then show 0 where the value was really unknown. There are no lists, nested values or binary fields.

File size limits

The headers of the .shp and .shx files store offsets and lengths as 32-bit numbers, which leads to the usual rule that each part of a shapefile, the .shp and the .dbf in particular, must stay below 2 GB. Some programs, GDAL among them, can read and write files up to about 4 GB when told to, but other software may reject them. A dataset larger than that has to be split into several shapefiles or stored in another format. In practice the .dbf often hits the limit first, because every text field takes its full width in every row, even when most values are short.

Text encoding and the .cpg file

The .dbf format predates Unicode. Text is stored as bytes, and the header has one byte, often called the language driver ID, that can name a code page such as Windows-1252 or a DOS code page. Many programs leave it empty or set it wrongly. The .cpg file was added later to state the encoding clearly, and modern software writes UTF-8 together with a .cpg that says so.

When neither the .cpg nor the header byte is right, a program has to guess, and names like Malmö, Ærø or São Paulo turn into strings of odd symbols. If you see that, the data is usually intact. Tell your GIS program which encoding to use when opening the layer, for example in the layer properties in QGIS, or add a .cpg file containing the correct name. When you write shapefiles yourself, choose UTF-8 and keep the .cpg with the other parts.

Coordinate systems and the .prj file

The .shp file stores plain numbers without saying what they mean. They might be longitude and latitude in degrees, metres in a UTM zone or feet in a US state plane system. Only the .prj file tells programs which coordinate reference system applies, and it does so as WKT text rather than an EPSG code. Most software can match common definitions to an EPSG code, but unusual or hand-edited .prj files sometimes lead to a slightly different interpretation.

Without a .prj, a program may assume WGS84 or ask you. Values between -180 and 180 and between -90 and 90 are probably degrees; large values in the hundreds of thousands or millions are projected metres or feet. Assigning a coordinate system only labels the data. Reprojecting actually recalculates the coordinates, and the two should not be confused.

Alternatives to the shapefile

GeoPackage is an open standard of the Open Geospatial Consortium, built on an SQLite database in a single .gpkg file. It holds several layers, long field names, proper dates and times, null values and large datasets, and it is the default format of QGIS. It is a good choice for handing over a whole project. GeoJSON is a plain-text JSON format standardized as RFC 7946. It is easy to read in a text editor and the natural format for web maps and APIs, but the standard expects WGS84 coordinates and the files get large. FlatGeobuf is a newer single-file binary format with a built-in spatial index, designed for fast reading and for streaming over the web; support is growing but is not yet as universal as for the others.

None of these is right for every case. The shapefile remains the safest choice when you do not know what software the recipient uses, or when a system or a client explicitly asks for it. When its limits get in the way, such as truncated field names, lost timestamps or files close to 2 GB, converting to GeoPackage is usually the least painful move.

Work with shapefiles in your browser with FileReshape

FileReshape's shapefile tools read the parts with GDAL compiled for the web, so the files are processed on your own device and are not uploaded. The shapefile viewer shows the features on a map and the attribute table, with the type of each field. The Shapefile to GeoJSON, Shapefile to KML and Shapefile to GeoPackage converters write the same data in those formats; GeoJSON and KML are written in WGS84, and the GeoPackage keeps the coordinate system. KML to Shapefile goes the other way and writes a .zip with .shp, .shx, .dbf, .prj and .cpg files, one shapefile per geometry type. You can drop a .zip or select the parts together, and names already shortened to 10 characters cannot be restored by any conversion.