Can I portray millions of geometries in my web client?

Let’s say you have an area composed of 1 million geometries. You are not always going to show everything. You pick and choose what you want to present depending on the zoom levels. This can be tricky because you also need to process the attribute data from the contained geometries (e.g. adding, averaging). There is this kind of illusion that the user is getting all the data all the time. The magic is done via a smart setup on the backend to minimize the features being retrieved.  Several approaches to tackle this issue:

1) Reduce the number of features being returned. You want super fast performance, present maximum 1000 features.

Pre-configure and select what to show at different levels (e.g. via style optimizations). As the user zooms-in, the data presented changes. New features and new labels might show-up, while others disappear.

Create bigger geometries that are composed of smaller geometries. The geometries returned depend on the zoom level of the request or the information of the request. When done properly discrete grids are an incredible mechanism to compute fast analytics. A good read about this topic is the  OGC Discrete Global Grid System (DGGS). There is also related work going on, which we are helping to advance, as part of the  OGC Testbed 16.

2) Simplify the vertices of the polygon. For example, if a polygon has 500,000 vertices at a higher zoom level the same polygon can be represented in hundreds of vertices and the end user should not notice the difference. The tutorial for using GeoTools Feature-pregenralized module explains this in detail.

Can I serve petabytes of raster data with GeoServer?

The answer is yes, but you need to properly configure them. Regarding formats GeoTIFF is the champion. It is very flexible, can be tiled and can be fined tuned for performance.  

How do you structure a GeoTIFF? There are several structures such as structuring the data in a Single GeoTiff, in a Mosaic or in a Pyramid.

  [caption id="attachment_5302" align="aligncenter" width="936"] GeoTIFF Structures[/caption]

The structure strategy basically depends on the size of the data, and the dimensions. 

• If single granules are < 20 Gb, using Single GeoTiffs are a good choice.
• If files are > 20 Gb or you have too many dimensions (e.g. numerical models might have multiple times, elevation and others), then use ImageMosaics.
• If the dataset if tremendously large and the data needs to be served at different resolutions, then ImagePyramid is the best option.

With the above approaches we have served satellite imagery covering the entire planet, as well as long time series of meteorological models, and the like, up to several petabytes of backend data.

As said before, I have only provided some tips and tricks in this post. Don’t miss our webinar and register, so you will have the opportunity to hear more about this topic and ask anything you want to Andrea.

Hope to see you virtually on June 10th, meanwhile stay safe and keep strong!

Cordially,

As most of you readers are aware, GeoServer is a very  popular open source tool to publish geospatial data. We, at GeoSolutions, are very fortunate to have in our team key code contributors, as well as, customers that support the advancement of GeoServer for the greater benefit of the geospatial community. Version 2.17 was released a week ago. A blog with detailed information is available at the GeoServer blog. In this post, I highlight some new functionality and invite you to a webinar on May 11th to learn more about the release from our GeoServer lead developer, Andrea Aime,

The GeoServer 2.17 release comes with many interesting new functionalities, let us briefly introduce some of them.

Faster, due to improvements in GeoWebCache

You will experience a much better startup performance because of the improvement to load the tiles layers configuration. Now, you can have  better control over failed seeding operations, by configuring error tolerance on single threads and across seed jobs (enabling seeding threads to continue if they encounter failures, like it occurred in the past).

Ability to render vector tiles with Mapbox Styles

You can now configure GeoServer to serve layers as vector tiles which can be used as sources for Mapbox styles rendered by web applications such as OpenLayers.

Improved the ability to read and write MBTiles files in GeoServer

Now you can read and write MBTiles files in GeoServer. You can use MBTiles as a raster input datastore as well as an WMS GetMap output format.

And more functionality like:

• Added support for FlatGeoBuf (incredible binary encoding for fast transfers)
• Added functionality to view and manage resources in the data directory (e.g. configuration files, icons and fonts)
• Better manage security for layers when looking at layer
• Ability to view when a layer was created and last modified
• Improved the rendering of icons and maps
• Enable changing background in the SLD
• Enable ability to enable/advertise layer groups
• Ability to add custom dimension in vector layers (similar to time and elevation, dimensions in WMS)
• Added more capabilities to WMS Cascading (e.g. specify format, min/max scale denominators for cascaded layers)
• Added support for JSON-LD
• Added support for OpenAPI
• And much more..

We cordially invite you to join the webinar on Monday May 18th at 11:00 EDT / 3:00 pm GMT (check more time zones), to learn more about this release from our GeoServer Technical Lead, Andrea Aime. We are planning to do 30 minutes presentation and 30 minutes for questions.  You can register by clicking on the button below.

Hope to see you virtually on May 18th, meanwhile stay safe and keep strong!

Cordially,

Sharing of health data has become one of the most important data sharing activities in the last months. The general population and authorities need to know the current situation not only about reported cases, deaths and recovered, but other data (e.g. hospital beds, ventilators, etc.) that supports properly responding to the COVID-19 pandemic. In addition to this health data, there are other non-health datasets (e.g. boundaries, socio-economic data) that provide context, complement and help decision makers better understand the possible actions to take. So, in today’s health emergency situation, an information sharing infrastructure with modern web visualization tools is critical to guide the way we share health and complementary data. 

In this blog I am going to 1) highlight the importance of a Health Spatial Data Infrastructure (SDI) documented in a recent paper published by OGC, which I co-authored, 2) make the case that sharing of data should be easy, and 3) provide an example of a dashboard tracking COVID-19 data based on MapStore.

The Open Geospatial Consortium (OGC) published some weeks ago the white paper: Health Spatial Data Infrastructure: Application Areas, Recommendations, and Architecture. It provides reference to initiatives (e.g. EO4Health, Eo2Heaven, INSPIRE Human Health and Safety Specifications,)  that have occurred in the last years related to building an infrastructure that enables the collection, exchange, integration, analysis, and visualization of health and non-health data to help identify and address health issues at global and local level. The sections I consider worth reading are: Section 4.7. Pandemic Response,  Section 5 Data Considerations and Related Recommendations, and  Section 6 - Health SDI Architecture Framework.

Section 4.7 discusses the development and use of different indexes (e.g. Transmission Risk Index) that are only possible to develop if there is enough information available. This is the reason why a Spatial Data Infrastructure is critical. This is the reason why we need to publish the data on the Web using open standards, following recommendations from Section 5 that takes into account security, anonymization and suggested formats and interfaces (e.g. WFS or WMS). 

Section 6 provides a discussion on the architecture and also a workflow about what is needed when creating indexes that involve different types of data (See Figure 1). What is critical from the workflow is the catalog, which is the heart of an SDI. If you don't know what data is available you can’t put it together in a fast manner and it requires phone calls, emails, searching, scraping websites, etc., wasting precious time when building a common operational picture.

Organizations harvesting information from different sources to create dashboards mostly rely on personal communications and getting the data from official web reports available at government/intergovernmental websites. Then, they create “machine readable formats” such as JSON or CSV that are ingested in the web clients. The “manual process” of getting the data requires a lot of human power, and fortunately for this crisis there are a lot of people willing to help. This is not the ideal. Government and other official sources should be making data available via open standards following the recommendation in the report.

At GeoSolutions, we were investigating how COVID-19 data was being published. We found an initiative, the COVID Tracking Project that actually does a great job on aggregating data for the US states. They perform quality control, publish the provenance, and make the data available via a simple Data API. If organizations around the world can follow simple APIs like the one mentioned, it will be easier to aggregate data in our current and other health emergency events. Also, fortunately, OGC is working on simple APIs that will enable government, agencies and projects (similar to the COVID Tracking Project), to share data via an open-standard simple API

Based on the previous mentioned project, and the simplicity of their API, we wanted to develop a simple sleek dashboard for tracking COVID-19 cases in the US. We wanted to reuse the best tools that we have in house at GeoSolutions, so we reused MapStore to build such a dashboard; you can reach it live here.

The client interacts directly with the API. There is no database on the backend and this is the reason it can be installed in a S3 bucket (which is a cheaper option than using a virtual machine under EC2). The user interface allows to view more than one variable at a time by enabling the user to select from the left side and displaying bubble plots located at the centroids of each state. The user can also view and order the statistics on the right column enabling easy comparison among the states. Hovering over a state shows all the data for that particular state, as it is shown for New York . Also, in our spirit of open source advocates, we have released the code at GitHub under a BSD License.

Sharing data via open standards is important, as well as, simple APIs that can enable web clients to provide dashboards and critical information to end users. I hope you find this blog useful and can inspire some ideas about sharing and visualizing pandemic tracking.  If you would like GeoSolutions to help with your health data publication, cataloging or visualization needs we are here to help!

Please don’t hesitate to contact us. We would love to hear how we can help you!

Cordially,

Open Street Map (OSM) is a popular collaboration mapping project that has over 2 million registered users. It provides the capability to update, using the power of crowdsourcing, critical information that can help decision makers and citizens to make day-to-day decisions. For example, the Humanitarian Open Street (HOT) Team is coordinating updating the status of hospitals, times that businesses are operating, and other useful data, during the COVID-19 outbreak. It is safe to say that OSM data is successfully used as base maps in a wide variety of tools and services.

When it comes to using OSM data in our maps there are many options available. These are some of them:

• You can use the main openstreetmap.org tile server
• You can set up your own service similar the  openstreetmap.org tile server
• You can use third-party OSM tiles providers
• You can set up an OGC server

 The key factors to account for when deciding how to serve OSM data are, in our opinion, the following:

• Production Readiness: as in the ability to use the tiles in operational environments, where an SLA might need to be ensured.
• Custom Tile Grids: as in the ability to customize tile gridsets to support more coordinate reference systems, different tile sizes, etc.
• Custom Styling: as in the ability to customize quickly the rendering styles to accustom different needs (e.g. land vs marine use case).
• Interoperability: ability to serving data via open standards (e.g. OGC WMS, WMTS or WFS) to enable easy use by most GIS clients and reduce vendor lock-in.
• Offline Use: as in the ability to use tiles offline or in protected environments (e.g. defense, security, emergency rescue, etc.)
• Usage Fee: as in the eventual per use costs to pay for using a tile service.

A few observations:

• Production readiness can vary a lot depending on your requirements, as well as, on the expertise of those who would be setting up the services.  The main openstreetmap.org site might not be the best choice for high traffic sites due the fairness restrictions which could, obviously, limit traffic.
• Setting up and maintaining your own stack does not come for free as it requires both expert time as well as hardware and software resources.
• Usage Fees can vary a lot between different services, depending on your needs and web traffic.
• If you need to use OSM data in a disconnected environment, the only options you have is to set up and host your own server.

Long story short, the number of options available is big, hence you need to analyze your use case in detail before making a decision, considering in the picture also your budget and available expertise.

As you might know, at GeoSolutions we have extensive expertise when it comes to GeoServer. While we are eager users of the original OSM service, we have found ourselves many times in the position to setup an OGC Server on top of OSM data via GeoServer for us and for our clients. Last month we published a blog about the process we went through to use GeoServer to publish and style OSM data. We also made available the GeoServer data directory containing the configuration and styles used to portrait the data.

Based on comments and popularity of the blog, we decided to setup a series of  free webinars to go through the whole process that goes from downloading OSM data to serving it as scale with GeoServer while we will also allow participants to interact with key GeoServer developers and DevOps at GeoSolutions. Therefore, we cordially invite you to join the first webinar with our GeoServer Technical Lead Andrea Aime, Serving OSM Data with GeoServer - Advanced Styling, which will take place on the 27th of April from 5pm to 7:30pm CET (that is from 11am to 1:30 pm EST or from 8am to 10:30 am PDT); you can register by clicking on the button below.

Once you register we will provide more information about how to download the data and other tools that we will use at the webinar (e.g Docker container, sample styles and sample of high resolution OSM data).

Hope to see you virtually on April 27th, meanwhile stay safe and keep strong!

Cordially,

Dear Reader,

We are pleased to announce the release 2020.01.01 of MapStore, our flagship Open Source webgis product. The full list of changes for this release can be found here, but the most interesting ones are described here below.

Important changes for this release

• Dashboard filtering capabilities: dashboards have been improved by including the possibility to define quick filters across different widgets. The online documentation is available here
• Various enhancements: the MapStore codebase has been improved to enhance WMTS support with Openlayers, the build process and the general error handling 
• Hardening: various bug fixes have been provided, the complete list is available here

Let us now go into more details about the new dashboard filtering capabilities. MapStore dashboards have been enhanced with the ability to filter widgets contents on the fly. The example here below show a simple scenario case built on top of data coming from one of our clients, special tanks to the Municipality of Genova for this. This dashboard shows the evolution of the Public Works in the city over time. The dashboard is also available in our MapStore live demo here (you can login with user demo  and password  123456).

With this you can use the Table widget in order to filter other widgets connected to it. The Table widget for dashboards has been improved with a quick filter tool that is available for each attribute and allows you to filter records inside the table and connected widgets that are using the same dataset. The Quick filter automatically is enabled as soon as a widget is connected to the Table one.

More details on how this new tools works, can be found inside the MapStore online documentation.

Ongoing and future work

For the next major release we will provide the best!

Application Contexts and Viewer Composer: as we anticipated in the previous blog post, an additional interesting feature will make MapStore even more powerful than now. The next major release (2020.02.00) will bring the new Application Context tool for MapStore that will allow administrators to define multiple viewer contexts in a three steps wizard. We made really interesting progress in the last couple of months, stay tuned!

Storytelling tool: we made some fantastic progress also for the new geospatial storytelling tool that will be available in MapStore starting from the next major release: we named this new tool GeoStory and it is expected to be released for the next major release (2020.02.00) of MapStore. Below a quick preview of the work.

Stay tuned for additional news on the next features! If you want to play with the current release, you can access the live demo here and login with username demo and password 123456.

If you are interested in learning about how we can help you achieving your goals with open source products like GeoServer, MapStore, GeoNode and GeoNetwork through our Enterprise Support Services and Subscription Services offerings, feel free to contact us!