Two trends are set to dramatically disrupt the long prevailing government and corporate data secrecy in the Middle East: data democratization through the web, and decreasing cost in collecting, analyzing, and publishing data for the masses.

These trends bring the promise of greater transparency and accountability, but do not come without their challenges, most notably in terms of increasing telecommunication access and citizen awareness, and establishing lawful oversight to maintain the balance between data openness, citizens’ rights, and national security.

The internet is changing previously held assumptions about data secrecy in the Middle East. (Image via Ero Electronic)

Compared with other countries, governments in the Middle East and North Africa generally maintain limited numbers of open databases to the public.

Businesses are equally protective mainly because private ownership (as opposed to stock market listing) is largely prevalent in the region meaning that many companies are not legally required to share information about their performance and other corporate activities.

Given the volatile security situations in some countries, governments impose secrecy and limit access to data. The result is that data is not as abundantly available, or accurate, as one may hope.

Zawya, an online provider of business information on the Middle East and North Africa, has managed to identify and monetize that gap by creating research teams that produce research reports mainly pertaining to the construction industry and businesses in the region. Other information providers with a similar focus remain rare. The main sources of detailed data are typically analysis companies in the US or Europe (e.g., Bloomberg, IHS) who produce expensive reports that cater to business executives and high level decision makers.

But all this may experience some change because of the internet. The increasing prevalence of information exchange through social media networks, and crowdsourcing (e.g., forums and blogs) is creating unprecedented amounts of data that includes news, opinions, and analysis, among others. While these large online and publicly available data sets may not be game changers on their own, the rise of elaborate and low cost data mining and analysis tools may pave the way to a significant shift.

Contrary to the established practice of setting up human research teams to collect, analyze and summarize data findings, increasingly capable and available software (a good part of which being open source) now allows the automatic creation of accurate structure from the seemingly unstructured maze of online data, and they do so at a fraction of the cost.

Unsurprisingly, entrepreneurs are riding the wave. Companies such as Dataminr, Salesforce, and iTrend, among many others, have built automated software solutions that tap into online data to identify noteworthy trends relating to the news, economic activities, customer sentiments, etc.

Other firms are attempting to build databases about regions which lack it or are unable to maintain it. One example is ACLED which stands for Armed Conflict Location & Event Data Project and tracks reported conflict and political violence with a particular focus on Africa.

Another example is Eqlim, a Beirut based startup that mines the public Internet to monitor the impact of geopolitics on economic activities across the Middle East.

Crowdsourcing solution providers with relevant activities include First Mile Geo which has partnered with Caerus Associates to map the conflict in Aleppo by using the web to survey people who are witnessing the events first hand, and the Syria Tracker Crisis map project.

Data is helping entrepreneurs, scientists, government agencies and think tanks better understand situations in the Middle East and around the world. (Image via Computer Weekly)

While these changes undoubtedly bear the promise of more transparency, and eventually accountability, they do not come without their challenges.

Internet access and technology literacy, particularly in some of the less developed countries, stand as one obstacle that may undermine the benefits of open data solutions altogether. Iraq for example has an internet penetration at around eight percent, Yemen and Syria at below 15 and 23 percent respectively, and many of the other countries in the region score relatively low in terms of household internet penetration rates.

But perhaps more worrying is the strong appetite of intelligence services, governments, and similar entities for this sort of information. The ramifications of the US National Security Agency’s PRISM program are still front news to remind us of how serious governments are about collecting and analyzing data.

As it were, the trends we have identified above serve as golden channels for those purposes. At the tip of the iceberg are companies such as Recorded Future and Palantir (a $5 billion startup funded by the CIA) which offer services that take in extremely large datasets and use them to provide insights to government agencies about aspects such as terrorism and cyber threats.

Change is upon us and history has shown that technological advances are almost impossible to halt. The solution to the challenges that accompany these technologies does not lie in restricting access to online services, like the Turkish government has recently tried to do; nor does it lie in hoping governments and other stakeholders will not tap into any useful data they can put their hands on.

One part of the solution lies in developing adequate telecommunications infrastructures to channel this information to the masses, while developing awareness by the average citizen about the value and relevance of such knowledge. And another part involves establishing proper governance for data mining and access.

Canada for example maintains a Privacy Commissioner whose main role is to safeguard citizens’ privacy, and the European Union’s recent ruling that has established the right of people to be “forgotten” by Google stands as a solid example of the need for, and effectiveness of proper oversight.

This is not to say that Western states are perfect role models (the PRISM scandal stands to remind us of that), but as far as the Middle East is concerned, there certainly is a need to setup proper frameworks for data access and analysis thereof that will embrace the tremendous opportunity for greater information that these technologies bring, without providing ways for governments to invade individuals’ lives and unlawfully tighten their grip on the societies they govern.