brails.utils.arcgisapi_service_helper module

This module defines a class for retrieving data from ArcGIS services APIs.

ArcgisAPIServiceHelper(api_endpoint_url)

A helper class for interacting with an ArcGIS API service.
class brails.utils.arcgisapi_service_helper.ArcgisAPIServiceHelper(api_endpoint_url)

Bases: object

A helper class for interacting with an ArcGIS API service.

This class provides methods to query the service, fetch element counts, split polygons into cells, and categorize cells based on their element count.

This class allows users to interact with an ArcGIS API endpoint to retrieve information about elements within specific polygons. It supports functionality for:

  • Retrieving the maximum number of elements that can be returned per query.

  • Categorizing and splitting polygon cells based on the element count.

  • Dividing polygons into smaller rectangular cells to ensure a balance of elements per cell.

  • Fetching specified attribute data for given cells in parallel.

The class includes methods for making API requests with retry logic, handling transient network errors, and parsing the responses to determine element counts within specific regions.

To import the ArcgisAPIServiceHelper class, use:

from brails.utils import ArcgisAPIServiceHelper
Parameters:

  • api_endpoint_url (str) – The base URL of the ArcGIS API endpoint used for requests.

  • max_elements_per_cell (int) – The maximum number of elements the API allows per query, fetched from the ArcGIS service.

Note

  • The class assumes a uniform distribution of elements across polygons when splitting.

  • The retry strategy uses exponential backoff to handle transient network issues.

  • The method get_element_counts provides a count of elements within the bounding box of a polygon, which is essential for categorizing and splitting polygons.

Raises:

  • ValueError – If an invalid Polygon or parameter is passed to a method. NotImplementedError: If the requested functionality is not supported by the API.

  • HTTPError – If the HTTP request fails after retries.

categorize_and_split_cells(preliminary_cells)

Categorize/split a list of polygon cells based on their element count.

This method processes a list of polygons (preliminary_cells) by first obtaining the number of elements contained within each polygon. If a polygon contains more elements than the specified maximum allowed per cell (max_elements_per_cell), the polygon is split into smaller cells. The method returns two lists:

  • A list of cells that are kept as is (those that do not exceed the element threshold).

  • A list of split cells (those that exceeded the element threshold).

Parameters:

preliminary_cells (list[Polygon]) – A list of Shapely Polygon objects representing the preliminary cells to be processed.

Returns:

A tuple containing two lists of Shapely Polygon objects:

  • The first list contains the cells to keep (those with number of elements <= max_elements_per_cell).

  • The second list contains the split cells (those with number of elements > max_elements_per_cell).

Return type:

tuple

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>> api_endpoint = (
...     'https://services1.arcgis.com/Hp6G80Pky0om7QvQ'
...     '/arcgis/rest/services/Transmission_Lines_gdb/'
...     'FeatureServer/0/query'
... )
>>> api_tools = ArcgisAPIServiceHelper(api_endpoint)
>>> cell = box(-81.334, 37.299, -71.908, 40.295)
>>> cells_to_keep, cells_to_split = api_tools.categorize_and_split_cells([cell])
>>> print(f"Cells to keep: {len(cells_to_keep)}")
Cells to keep: 0
>>> print(f"Cells to split: {len(cells_to_split)}")
Cells to split: 21
download_all_attr_for_region(region, plot_cells=False, task_description='Obtaining attributes for each cell')

Download all attribute data for the specified region.

This method:

  • Retrieves the boundary polygon of the region.

  • Splits region into smaller cells for manageable data querying.

  • Refines the mesh by recursively splitting oversized cells.

  • Optionally plots the final mesh.

  • Downloads data (e.g., bridge attributes) for each cell using a provided data-fetching function.

Parameters:

  • region – A Region object that provides a get_boundary() method returning a polygon, region name, and optional OSM ID.

  • plot_cells (bool, optional) – If True, generates and saves a visualization of the final meshed cells.

  • task_description (str, optional) – A message string that describes the task being performed, passed to the data-fetching method for logging or display.

Returns:

  • downloaded_data (Dict[Polygon, List[Dict[str, Any]]]):

    Mapping from each final mesh cell polygon to a list of attribute dictionaries (e.g., representing bridges or other assets).

  • final_cells (List[Polygon]):

    List of polygons representing the final meshed cells used for data querying.

Return type:

Tuple

Example

>>> from brails.utils import ArcgisAPIServiceHelper, Importer
>>> importer = Importer()
>>> region_boundary_class = importer.get_class('RegionBoundary')
>>> region_boundary = region_boundary_class(
...         {'type': 'locationName', 'data': 'Los Angeles, CA'}
...     )
>>> api_endpoint = ('https://services5.arcgis.com/7nsPwEMP38bSkCjy'
...     '/arcgis/rest/services/Building_Footprints/FeatureServer'
...     '/0/query')
>>> api_tools = ArcgisAPIServiceHelper(api_endpoint)
>>> (
...     downloaded_data,
...     final_cells
... ) = api_tools.download_all_attr_for_region(
...     region_boundary,
...     plot_cells=False
... )
Searching for Los Angeles, CA...
Found Los Angeles, Los Angeles County, California, United States
Meshing the defined area...
Obtaining the number of elements in each cell:
100%|██████████| 661/661 [00:07<00:00, 89.60it/s]
Obtaining the number of elements in each cell:
100%|██████████| 2576/2576 [00:27<00:00, 92.85it/s]
Obtaining the number of elements in each cell:
100%|██████████| 1146/1146 [00:11<00:00, 97.85it/s]
Meshing complete. Split Los Angeles into 3824 cells.
Obtaining attributes for each cell:  100%|██████████| 3824/3824
[02:46<00:00, 22.93it/s]
>>> total_assets = sum(map(len, downloaded_data.values()))
>>> print(f'Total number of assets: {total_assets}')
Total number of assets: 1282028
download_all_attr_from_api(cell)

Download attribute data for a given cell using the ArcGIS API.

This method queries the ArcGIS API for all available attributes within the specified polygon cell.

Parameters:

cell (Polygon) – A Shapely Polygon object representing the geographic cell for which to retrieve bridge data.

Returns:

A list of dictionaries, each representing the attribute data for a bridge within the cell. Each dictionary includes geometry and attributes keys as returned by the ArcGIS API.

Return type:

List[Dict[str, Any]]

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>> api_endpoint = (
...     'https://services5.arcgis.com/7nsPwEMP38bSkCjy'
...     '/arcgis/rest/services/Building_Footprints/FeatureServer'
...     '/0/query'
... )
>>> api_tools = ArcgisAPIServiceHelper(api_endpoint)
>>> cell = box(-118.244, 34.041, -118.243, 34.041)
>>> data = api_tools.download_all_attr_from_api(cell)
>>> print(data)
[{'attributes': {'OBJECTID': 385248, 'CODE': 'Building', 'BLD_ID':
'487907837328', 'HEIGHT': 37.14, 'ELEV': 292.15, 'AREA': 9665,
'LARIAC_SOURCE': 'LARIAC2', 'LARIAC_DATE': '2008',
'AIN': '5147028043', 'STATUS': 'Unchanged', 'CODE_NUM': 1},
  'geometry': {'rings': [[[-118.24332934719, 34.0407349647641],
     [-118.243413068638, 34.0406417365415],
     [-118.243415138653, 34.0406430242604],
     [-118.243416898264, 34.0406440782071],
     [-118.243682323766, 34.0408091186429],
     [-118.243541516086, 34.04096427197],
     [-118.243533076239, 34.0409678617538],
     [-118.243522111045, 34.0409659050115],
     [-118.2434982541, 34.0409913987935],
     [-118.243502997061, 34.0409990828814],
     [-118.243501803082, 34.0410085657733],
     [-118.243492197701, 34.0410198786033],
     [-118.243226233836, 34.0408560099184],
     [-118.24332934719, 34.0407349647641]]]}}]
download_attr_from_api(cell, requested_fields)

Download specified fields from the API for a given cell.

Parameters:

  • cell (Polygon) – A Shapely Polygon object representing the area of interest.

  • requested_fields (list[str] or str) – A list of attribute names or the string ‘all’.

Returns:

A list of features (attributes) fetched from the API.

Return type:

list[dict]

Raises:

ValueError – If the cell input is not a valid Polygon or requested_fields is not valid.

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>> api_endpoint = ('https://services5.arcgis.com/7nsPwEMP38bSkCjy'
...     '/arcgis/rest/services/Building_Footprints/FeatureServer'
...     '/0/query')
>>> api_tools = ArcgisAPIServiceHelper(api_endpoint)
>>> cell = box(-118.244, 34.041, -118.243, 34.041)
>>> data = api_tools.download_attr_from_api(
...     cell,
...     ['HEIGHT', 'ELEV']
... )
>>> print(data)
[{'attributes': {'HEIGHT': 37.14, 'ELEV': 292.15},
  'geometry': {'rings': [[[-118.24332934719, 34.0407349647641],
     [-118.243413068638, 34.0406417365415],
     [-118.243415138653, 34.0406430242604],
     [-118.243416898264, 34.0406440782071],
     [-118.243682323766, 34.0408091186429],
     [-118.243541516086, 34.04096427197],
     [-118.243533076239, 34.0409678617538],
     [-118.243522111045, 34.0409659050115],
     [-118.2434982541, 34.0409913987935],
     [-118.243502997061, 34.0409990828814],
     [-118.243501803082, 34.0410085657733],
     [-118.243492197701, 34.0410198786033],
     [-118.243226233836, 34.0408560099184],
     [-118.24332934719, 34.0407349647641]]]}}]
static fetch_api_fields(url)

Fetch the list of attribute names (fields) from an ArcGIS REST layer.

Parameters:

url (str) – The URL of the ArcGIS layer endpoint (can include '/query').

Returns:

A list of field names defined in the layer.

Return type:

list[str]

Raises:

  • KeyError – If the 'fields' key is not present in the response.

  • requests.RequestException – If the request fails.

Example

>>> from brails.utils import ArcgisAPIServiceHelper
>>>
>>> api_endpoint = (
...     'https://hazards.fema.gov/arcgis/rest/services/FIRMette'
...     '/NFHLREST_FIRMette/MapServer/20/query'
... )
>>> field_names = ArcgisAPIServiceHelper.fetch_api_fields(
...     api_endpoint
... )
>>> print(field_names)
['OBJECTID', 'DFIRM_ID', 'FLD_AR_ID', 'STUDY_TYP', 'FLD_ZONE',
'ZONE_SUBTY', 'SFHA_TF', 'STATIC_BFE', 'V_DATUM', 'DEPTH',
'LEN_UNIT', 'VELOCITY', 'VEL_UNIT', 'AR_REVERT', 'AR_SUBTRV',
'BFE_REVERT', 'DEP_REVERT', 'DUAL_ZONE', 'SOURCE_CIT', 'SHAPE',
'SHAPE.STArea()', 'SHAPE.STLength()', 'GFID', 'GlobalID']
fetch_data_for_cells(final_cells, download_func, desc='Obtaining the attributes for each cell')

Download data for a list of cells using the provided function.

Each cell in final_cells is processed concurrently using a thread pool. Results are stored in a dictionary mapping each cell to its downloaded data. If a cell’s download fails, the value will be None.

Parameters:

  • final_cells (List[Any]) – List of cells to process.

  • download_func (Callable[[Any], Any]) – Function to download data for a single cell.

  • desc (str) – Description for the progress bar.

Returns:

Dictionary mapping each cell to its downloaded data (or None if failed).

Return type:

Dict[Any, Any]

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>> api_endpoint = (
...     'https://services1.arcgis.com/Hp6G80Pky0om7QvQ/'
...     'arcgis/rest/services/Transmission_Lines_gdb/'
...     'FeatureServer/0/query'
... )
>>> api_helper = ArcgisAPIServiceHelper(api_endpoint)
>>> cells = [
...     box(-86.940, 24.545, -77.513, 27.992),
...     box(-77.499, 38.779, -76.910, 38.966)
... ]
>>> results = api_helper.fetch_data_for_cells(
...     cells,
...     api_helper.download_all_attr_from_api
... )
Obtaining the attributes for each cell: 100%|██████████| 2/2
[00:00<00:00,  2.37it/s]
>>> for cell, data in results.items():
...     print(
...         f'Cell bounds: {cell.bounds}',
...         f'Number of assets: {len(data)}'
...     )
Cell bounds: (-77.499, 38.779, -76.91, 38.966)
Number of assets: 152
Cell bounds: (-86.94, 24.545, -77.513, 27.992)
Number of assets: 1376
fetch_max_records_per_query()

Retrieve the maximum number of records returned by the API per query.

This function sends a request to the specified API endpoint and parses the response to determine the maximum number of records that can be returned in a single query. If the API does not provide this information or returns a value of 0, it raises an error.

Returns:

The maximum number of elements the API allows per query.

Return type:

int

Raises:

  • ValueError – If the API returns a value of 0 for the maximum number of elements, indicating an issue with the API or its response.

  • HTTPError – If the HTTP request fails (e.g., due to connectivity issues or a server error).

Example

>>> api_tools = ArcgisAPIServiceHelper(
...     'https://sampleserver6.arcgisonline.com/arcgis/rest/'
...     'services/Census/MapServer/3/query'
... )
>>> max_records = api_tools.fetch_max_records_per_query()
>>> print(max_records)
1000

Note

  • The function expects the API to return a JSON response containing a maxRecordCount key.

  • A retry strategy is implemented for the HTTPS request to handle transient network issues.

get_element_counts(bpoly)

Get the count of elements within the bounding box of the given polygon.

Parameters:

bpoly (Polygon) – The polygon marking the boundaries of a region.

Returns:

The count of elements within the bounding box, or 0 if an error occurs.

Return type:

int

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>>
>>> api_endpoint = (
...     'https://services1.arcgis.com/Hp6G80Pky0om7QvQ/'
...     'arcgis/rest/services/Public_Schools/FeatureServer/0/query'
... )
>>> helper = ArcgisAPIServiceHelper(api_endpoint)
>>> # Create a polygon covering the specified bounding box
>>> cell = box(-86.940, 24.545, -77.513, 27.992)
>>> # Get the element count for that polygon
>>> count = helper.get_element_counts(cell)
>>> print(count)
2109
split_polygon_into_cells(bpoly, element_count=-1, plot_mesh='')

Divide a polygon into smaller cells based on its element count.

If the number of elements exceeds max_elements_per_cell, the polygon is split into multiple rectangular cells. The grid is generated under the assumption that elements are roughly uniformly distributed, so the method produces an approximate balance of elements per cell. This method does not guarantee that every cell will be under the maximum element threshold.

Parameters:

  • bpoly (Polygon) – The polygon to split into rectangular cells.

  • element_count (int, optional) – Total number of elements in the polygon. If not provided, the method will compute this using get_element_counts method.

  • plot_mesh (str, optional) – If provided, the generated mesh will be plotted using GeoTools.plot_polygon_cells and saved to this file path.

Returns:

A list of polygons representing the rectangular grid cells covering the input polygon.

Return type:

List[Polygon]

Note

  • If the element count is below or equal to max_elements_per_cell, the polygon’s envelope is returned as a single cell.

  • If the element count exceeds max_elements_per_cell, the polygon is split into smaller cells based on the bounding box aspect ratio.

Example

>>> from shapely.geometry import box
>>> from brails.utils import ArcgisAPIServiceHelper
>>> api_endpoint = (
...     "https://services1.arcgis.com/Hp6G80Pky0om7QvQ/"
...     "arcgis/rest/services/Public_Schools/FeatureServer/0/query"
... )
>>> helper = ArcgisAPIServiceHelper(api_endpoint)
>>> # Create polygon for bounding box
>>> cell = box(-105.638, 24.966, -67.933, 38.031)
>>> # Split polygon into smaller cells
>>> rectangles = helper.split_polygon_into_cells(cell)
>>> for rect in rectangles[:2]:
...      print(rect.wkt)
POLYGON ((-105.638 24.966, -67.933 24.966,
-67.933 26.153727272727274, -105.638 26.153727272727274,
-105.638 24.966))
POLYGON ((-105.638 26.153727272727274, -67.933 26.153727272727274,
-67.933 27.341454545454546, -105.638 27.341454545454546,
-105.638 26.153727272727274))