Back to latest

Examining procurement “red flags” in Latin America with data

For anyone interested in exploring how open data might help make public contracting more honest and efficient, Latin America is a great place to start. The region is home to almost half of the world’s OCDS publishers and some of the largest and most complete contracting datasets. And crucially, data quality is now high enough in several jurisdictions to be used for improving public procurement outcomes.

It’s impossible to manually track and monitor hundreds of thousands of contracting processes. Publicly-accessible open data can make the task manageable, more reliable and credible, when they’re used to develop machine analytics to detect contracting processes that require further human investigation. So early in June, procurement practitioners from across the region gathered in Santiago for a workshop on “red flags”, risk indicators that can help to identify potential wrongdoing or inefficiencies in contracting processes, such as:

The participants—who came from government, oversight authorities, civil society and media in seven countries (Argentina, Colombia, Peru, Honduras, Chile, Paraguay and Mexico)—shared approaches and tools to identify risks in procurement that affect public integrity and efficiency, and explored how to use Open Contracting Data Standard (OCDS) data to build reliable, low-cost red flag models, with tangible examples from several countries.

Building & testing simple red flag tools

In a hands-on data expedition adapted from the School of Data’s methodology, each group calculated meaningful red flags by exploring the OCDS data available from their own jurisdiction. They chose which red flag they wished to analyze and applied their mini-model to a small sample of data, which we had prepared for them in advance. These are the red flags they selected and how far they managed to get in half day:







A quick note on our methodology

Each group’s dataset consisted of a sample of 500 procurement processes (OCDS releases) from 2018, which were selected from the (often huge amounts) of data available in each jurisdiction. We used the OCDS flatten tool to convert each JSON file into two tabular files: one with field names in Spanish and the other in English. We also provided coverage tables that informed the users which fields were present in each dataset, and the share of procurement processes included that information. 

The teams consisted of a policy expert, data curator, data analyst, and a data storyteller, which leveraged the skills of our talented partners to analyze, design, develop, and implement mini-models. Our practitioners selected their preferred analysis tool to calculate the red flags, ranging from Excel spreadsheets to scripts in the programming language R or even Open Refine. It was encouraging to see how these flags can be calculated in practice, regardless of the technical level of the person performing the analysis.

The data expeditions took place on the second day of the workshop after we had introduced participants to red flag models used elsewhere (see OCP’s own work on red flags). Anastasiya Kozlovtseva shared TI Ukraine’s model, which analyzes 40 risk indicators supported by machine learning from the citizen-reporting platform Dozorro. As of June, it is used by almost a million users and has identified more than 200,000 violations to contracting rules. Anastasiya also explained their “Tinder for Tender” tool, a clever application of the model, which was used internally by procurement experts to classify suspicious contracting processes.

We showed how five of the most commonly used red flags can be calculated, using a Google collaboratory notebook and data from Colombia Compra Eficiente and ChileCompra, which we gathered with the OCDS Kingfisher tool. A great discussion started as we went through the results, gave context to the numbers and indicator thresholds, and identified similarities between the countries. What could be an indication of a risky process in one country, could be normal in another jurisdiction, while it may be outright banned elsewhere. 

The idea for this workshop emerged from our collaboration with Chilean civil society partners Observatorio del Gasto Fiscal and Chile Transparente, who have pushed from within ChileCompra’s civil society council (COSOC) for the use of open data for increased integrity. The government procurement agency ChileCompra, has also been supportive of the initiative, seeing the potential in having civil society lead the development of such a model.

What next for red flags in Latin America?

One of the big takeaways was how standardized data can streamline efforts from different places and share models and knowledge more effectively across countries wanting to monitor procurement indicators and identify red flags. 

Going through the data also helped identify possible mapping issues and quality problems: when looking at tender periods, some agencies in Colombia apparently open their tender to receive bids for less than one day, signaling a probable issue with how the data is being captured in the procurement platforms. 

Some other participants realized that data wasn’t available to calculate some of the red flags that were relevant for them, especially those related to the implementation stage, which is seldom published. All of these aspects will help inform improvements to data validation that should lead to better quality data. 

Against the backdrop of the gorgeous snow-capped Andes, we closed the workshop with participants making concrete commitments for further developments: Pablo Seitz, the director of Paraguay’s public contracting agency (DNCP) agreed to publish at least three red flags on their website for public monitoring; Mexican and Chilean representatives committed to form working groups to advance this subject in their countries, and Colombia pledged to use OCDS data for fiscal audit means. Buenos Aires and Honduras promised to build on their recent OCDS data publication to start setting up tools such as the ones discussed. 

We believe this could be the starting point to better track what’s happening through efficient red flag models in Latin America that can serve to transform public contracting. Our helpdesk is ready to answer all your questions, so don’t be shy and tell us what you think!

Related Stories