A lot is happening in the EAGE digital world: on this page you can find some highlights from the latest initiatives on machine learning, A.I. and digitalization involving EAGE members worldwide, in addition to new contributions on the topic of Artificial Intelligence by the EAGE A.I. special interest community every week.
Digitalization Opportunities and Resources
The EAGE Short Courses catalogue has a section entirely dedicated to Data Science for geoscientists and engineers interested in learning new skills. From elementary to advanced level, EAGE Education offers various opportunities to approach the world of digitalization.
The digital transformation means change in every industry and enterprise, for everyone. EAGE is proud to accompany this process by providing a platform for energy experts and data scientists to discuss challenges and solutions. Join the conversation at one of the following meetings:
- The First EAGE Digital Subsurface Conference in Latin America (26 – 28 May 2021)
- Workshop 16 at the EAGE Annual: Development of ML Solutions at Scale: Going from Proof of Concepts to Integrated Workflows (22 October 2021)
- EAGE Digital 2021 (18 – 21 October 2021)
- or browse for more in the EAGE calendar of events!
More information and updates are shared in the EAGE Digital Newsletter. Sign up to receive it in your mailbox every month!
Advice by the EAGE A.I. Committee
The A.I. Committee discusses tips, techniques, learning experiences and whatever else is of interest to stay informed and up to speed with this emerging field. By sharing weekly tips on A.I. they aim to support geoscientists maintain their employability throughout the many reorganization cycles when the world requires different skills.
|A.I. topic of the week||Advice by the EAGE A.I. Committee|
|MLP Mixer||Can we replace CNNs and Transformers with Multi-layer-Perceptrons?
Recently, several publications have proposed that the recent advances in computer vision which have been achieved through new convolutional neural networks and transformer networks could also be achieved by a set of multi-layer perceptrons trained on extremely large datasets.
Yannic Kilcher has created a very nice video explaining the MLP-Mixer publication:
What do you think: Will Transformers replace convolutional networks or is attention really not what we needed after all?
|Big Data Analytics Using Lazy Evaluation||Data wrangling refers to the practice of cleaning and shaping data. It typically requires multiple steps, such as removing or substituting missing values, renaming columns, changing data types, or creating new categories from existing ones. These steps are absolution crucial for machine learning as the data needs to be prepared before it can be analysed.
There are two approaches to data wrangling. One approach, used by Pandas (an exclusively Python library) is known as the ‘eager’ evaluation model. Eager means that each operation is applied immediately at the point of call. The drawback of this approach is that no optimisation on the data preparation process can be made since each step is executed independently. By contrast, libraries that use a ‘lazy evaluation’ approach build a computation graph, and only apply the operations of the graph once you ask to collect the data and after the library optimises the computation graph. The big benefit of this model is that optimisation in terms of memory and calculation efficiency can be made, for instance by splitting the calculation dynamically over several GPUs or CPUs, or reducing the number of slow operations such as copying data between different memory register or different GPUs.
Probably the best known library designed around lazy evaluation is Apache Spark, which is written in Scala but has API bindings in Python and other languages. Because lazy evaluation offers the ability to optimise computational speed and memory use, Apache Spark is the de-facto Big Data processing engine in the enterprise world. There are however newer libraries out there too, with some promising concepts and easy to use Python APIs. Py-polars is a library written in Rust but with python APIs, and it is heralded by some as the successor of Pandas. It is capable of doing both eager and lazy evaluation and its syntax is very similar to Pandas. So for your next project using big data, take a look at Apache Spark and Py-polar and see if it is time to move away from Pandas.
Refer this recent article for reasons to choose Spark over Pandas
Here is a good article on Py-polar
Read tthishis for a more in-depth view of lazy and eager (and even greedy) evaluation from a functional programming point of view
|Getting started with A.I.||Getting started is often the hardest thing. Fortunately, nowadays there are many sites out there to get you started. With open source data and code there really is no excuse not to get started. (yes, even in the oil & gas industry you can find open source software nd data). All you have to do is come up with a new idea (and learn to code). Here are some links to get you started
Open source software
Open source data
Example of open source seismic data processing, recently used in the 2nd EAGE Machine Learning workshop
|Using callbacks to dramatically improve the learning process in Neural Networks||Callbacks are functions used as argument for other ‘base’ functions to monitor and control their performance and outcomes without modifying the source code.
Application in the training loop of a Neural Network enables customized monitoring (i.e. log files) and automatic intervention, for instance hyperparameter modification, saving model parameters, early stopping, etc. This is done essentially using on the fly monitoring of selected metrics while comparing them to preset criteria.
One example is lowering the learning rate when the improvement of certain validation metrics drops below a certain value. In short, this ensures significant improvement in both efficiency as well as optimization of setting hyperparameters, as compared to manual ‘trial and error’ approach.
Refer to this link for a general discussion callbacks in Python
Here is a guide on the inbuilt callbacks in TensorFlow. Read more here.
|Computer vision||Enabling computers to see and understand images like human vision, computer vision is one of the most powerful types of AI and you’ve almost certainly experienced it every day. Recently, computer vision has been able to take big leaps driven by rapid development of AI (especially deep learning and neural networks), computing power and significant amount of visual data available. Popular computer vision tasks involve image classification, object detection, image segmentation, etc. Despite tremendous advances in real-world applications, helping computers to see turns out to be very challenging and we are still far from solving it. Are you keen to take your first step and contribute to its latest development?
Introduction to computer vision
Hands-on tutorial series
Stanford lecture collection
|Transformers for NLP and Image Generation||Transformers are a recent neural network architecture that has proven extremely successful in natural language processing (NLP) applications. This neural network is built around the concept of self-attention which Andrew Ng explains in one of his lectures.
Training of these NLP models is extremely demanding in terms of computational resources and can easily require thousands of GPUs for multiple weeks. Luckily there exist shared pre-trained transformer models for various NLP tasks provided by institutions like Huggingface and OpenAI.
Recently these flexible sequence-based models have been extended to the image domain leading to impressive results in image generation and classification.
|Physics-informed A.I.||Advances and breakthroughs in neural network and machine learning takes some time to be picked up. The geoscience community was a bit slow to pick up on the development in convolutional neural networks, but have fully embraced CNNs and U-net style networks in the last two years for all sort of interpretation problems. However, they are not the right tool for inversion style problems, simply because they lack knowledge of the physicsand hence have issues with generalization. Other fields have realized the same thing and hence the development of physics-informed or physics guided neural networks. This year we will see a lot of those
1) For a nice introduction see this site and its examples
2) An application to the geosciences
3) For those with more time on their hands, sit back and enjoy a full workshop
|Interpretable Deep Networks||A novel approach to make Deep Networks interpretable
It is (nearly) impossible to make direct observations inside the hidden layers (latent space). Standard DNN builds abstract features purely on statistical grounds and can be scattered; these are the main reasons for the ‘black box’ nature.
In a recent (Dec 2020) publication, Zhi Chen et al of Duke University propose Concept Whitening (CW), a modification to selected hidden layers such that they better represent known (sub) features up to that point in the network. The main idea of CW is ‘disentangle’ the latent space so different parts (layers) represent different (user defined) concepts.
To put it (overly) simplified, the network is then not only tuned using the main dataset, but also selected layers are tuned to sets of selected sub features.
Experiments with CNN networks so far have shown great promise.
For a clear(er) explanation
A copy of the paper
The code used
|What is A.I. good for and what is it not?||If we put “Hollywood AI” aside and look at practical applications of AI in an industrial context there is usually a sweet-spot were AI excels.
AI is not magic, it will typically not find patterns in data that you cannot find in other ways. If you have looked at a dataset over and over again don’t expect an AI system to suddenly find an answer that you couldn’t find before. AI systems are though VERY good at doing the same well-defined problem very quickly and millions of times. Use them for highly repetitive, high dimensional problems, e.g. find me similar music or seismic wavelets, find me anomalies in financial transactions or well logs (across thousands of datasets). Keep it simple and you have a good chance of success.
|A.I. in the Real World||Happy New Year 2021! This year my mission is to gain more hand-on experience with a wider range of data types and data science approaches. One of the hardest challenges when learning data science methodologies is bridging the gap between toy examples provided in courses to moving into real world, domain specific problems where you are not even sure if what you are attempting is possible with your given dataset. Kaggle datasets provide a great middle ground containing some relatively dirty datasets that range in application from fashion to academic performance to geoscience. One of the best things about these datasets, beside their open availability, is that users can upload their notebooks and have discussions around insight that can be garnered from these datasets. A few to get you going are:
Brent oil prices 1987-2020
Geology Image Similarity
Volcanic Eruption Prediction
|Generative Adversarial Networks||Generative Adversarial Networks (GANs), “the most interesting idea in the last 10 years in Machine Learning”, are a class of deep learning models capable of creating new data instances that resemble the training data. GANs can improve image resolution, augment training datasets, create A.I. art, etc. Within GANs architecture, two neural networks - a generator and a discriminator, are trained jointly with opposite goals: the generator learns to make fake data that look real to fool the discriminator, while the discriminator learns to distinguish the generated fake data from the real. Both networks are becoming better and better during the fight against each other, until the generator can produce realistic outputs given random inputs.
Google Machine Learning Crash Course on GANs (Beginner)
Coursera GANs Specialization (Intermediate)
The paper of Ian J. Goodfellow and co-authors first proposing GANs (Advanced)
|ML Libraries||Open-source software has been a key ingredient in the widespread adoption of machine learning technologies. Many libraries exist for different programming languages such as Python, R, or C++. In this weekly contribution we highlight a few of the well-known and upcoming machine learning libraries:
Scikit-Learn: This library is the bread and butter of all machine learning libraries. It contains not only implementations of many algorithms, but also supporting functionality for preprocessing, cross-validation, and metric-based evaluation.
Pytorch and Tensorflow: When a Random Forest won’t do the Job, these two libraries provide the necessary tools to build various deep neural networks. Both provide automatic differentiation capabilities and can scale from a laptop to HPC clusters.
PyMC3: For all things Bayesian modeling, this library provides all the necessary tools to build complex hierarchical models and allows for fast inference using modern implementations of Markov-Chain methods.
|GPT-3, the largest neural network in the world||GPT-3 has made the AI headlines since it appeared in May 2020. It is a product of the company OpenAI, and it can write poetry, translate, calculate, write code, have on-line conversations or write papers... It is the largest neural network in the world, with a total of 175 Billion parameters. GPT-3 was trained by reading 500 Billion words, that is the equivalent of 150 times the size of Wikipedia (in all the different languages)!
Wikipedia provides a general presentation of GPT-3.
There are plenty of different things that GPT-3 can do, many are useful and some are potentially harmful.
GPT stands for “Generative Pretrained Transformer“. GPT-3 addresses some of the well-known issues associated with standard Recurrent Neural Networks.
|A.I. Back to basics||Artificial Intelligence seems to be all about fancy machine learning and neural networks mathematics and algorithms. The reality is that easily 80% of the time will be spent getting your data ready for action. For geoscientist this at least is familiar, since before you can run your fancy RTM or seismic inversion, there is quite some pre-processing to be done also. So, this week we go back to some basics and since we like Python, that means Python basics.
1 page Pandas ccheat sheetheat sheet
Tutorials on various pre-processing topics
Complete course on Python for datascience
|Graph NN||Any data related problem statement can be represented using a Graph network, which is a mathematical construct defining interactions between data objects. Formally expressed as an ordered pair G of two sets V (vertices or nodes; data objects) and E (edges; interconnections): G = (V, E).
Graphs can have any structure; Decision Trees is an example of Graphs with extra restrictions on direction and connectivity.
Graph Neural Networks (GNN) is a category of learning methodologies for optimizing Graph networks currently under rapid development and showing high potential in effectiveness and efficiency.
For application of Graph networks to generate fast physics simulators, check this video.
Here is also an easy to read (re) introduction to Graph theory and a fairly readable short tutorial on GNN applied to Imaging with PyTorch examples.
|Vulnerability of NN||This week we will discuss the vulnerability of neural networks to hacking attempts either by manipulation from a software perspective or by altering input data in the physical world. Towards Data Science provides
a nice introduction to the security vulnerabilities of NNs and the different forms in which attacks can take.
The most common attacks are in the form of strategically adapting the input data which
fools the network into a misclassification. To the human eye, the adapted input data is
often almost identical to the original input data however these small adaptations have the power to completely deceive the classification procedure.
At a software level this can be by adding noise to the input data as illustrated by Goodfellow et al., 2015.
Their experiments showed how computationally-generated noise can be used to trick a network into misclassifying images that visually look identical, resulting in incorrect classifications with a very high confidence score.
Adverserial attacks can also be performed in the physical world by adding stickers or patches to objects to confuse classification network. Brown et al., 2018 illustrate the use of physical adverserial stickers
that when placed within a cameras reference frame cause misclassification of a banana
to be identified as a toaster.
|A.I. to enable the Energy |
|Across the world and throughout the energy industry the direction of travel is clear.
The world needs to dramatically cut emissions while ensuring there is enough energy for countries and communities
to continue to develop. AI will have a key role to play. Whether that be in terms of energy efficiency
and optimisation, reducing emissions, low carbon energy generation, energy distribution and storage.
Below are several views from different parts of the energy creation and consumption ecosystem:
Energy company view
Consulting company view
AI has a key role to play to enable a future more sustainable world.
People who can critically apply such techniques have a vital role to play in our future world.
|Cross-Validation for |
|Many predictive tasks we encounter in the subsurface are of spatial or temporal nature
e.g. predicting porosity and permeability away from well-control, or predicting the future flow behavior
of a subsurface reservoir given historical data.
In many applications, we evaluate the performance of algorithms using cross-validation.
Sebastian Raschka’s introduction to model validation provides an excellent overview of the definitions,
assumptions, and techniques used to choose the best algorithms and their parameters.
Code examples (Part IV) provide a practical starting point for practitioners.
Spatially correlated data used to build predictive models can have a significant impact on our ability
to judge the spatial predictive performance of algorithms and can lead to an optimistic bias in model evaluation. Roberts et. al. provide a comparison of various temporal, and spatial validation strategies as well as the significant
impact the choice of validation strategy can have on our ability to judge a model’s predictive performance.
Choosing the right validation strategy for the task at hand, allows practitioners to reduce bias
through model selection and builds trust in a method’s ability to make predictions away
from data and the future state of subsurface systems.
|Gaussian Processes and NN||Gaussian Processes (GPs) for Machine Learning are closely related to geostatistical models,
with the exception that Geostatistics tends to focus on one, two or three-dimensional models,
while GPs typically live in spaces of very large dimension. GPs are often used to generate
possible stochastic realizations constrained by data and provide a way to quantify uncertainties.
The book “Gaussian Processes for Machine Learning” by Rasmussen and Williams,
is a great introduction to GPs.
Neal showed that, before training, feed-forward Neural Networks (NNs) with just one infinite hidden layer,
generate a GP with a covariance derived from the NN’s activation function and the initial probability
distributions of the NN’s weights and biases.
Neal’s results have been generalized to deep and convolutional networks.
This means that, by defining a NN’s architecture and its hyperparameters, we are already defining
an implicit “prior” on the output of the NN. The concept of “Deep Image Priors” takes advantage of this
by proposing not to train the model using a Training Set, but to directly apply the prior NN model
to the optimization task. This has close links with Bayesian Deep Learning,
that we will discuss in the near future.
|A.I. failures||There are many inspiring quotes on failure. Like Thomas Edison’s “I have not failed.
I've just found 10,000 ways that won't work” and Churchill’s “Success is not final, failure is not fatal:
it is the courage to continue that counts.”. Most of the quotes encourage one to persists and
to take lessons from the failed endeavors. Failures in A.I. and machine learning happen all the time,
they are just not talked about much and therefore such learning is not as easy to come by
as the learnings from success. Here are some links about failure,
overpromise and underdelivery of A.I. and machine learning technology for you to learn from.
1) Weapons of Math Destruction
2) How IBM Watson Overpromised and Underdelivered on AI Health Care
3) Consumer Reports Unmasks Tesla’s Full Self-Driving Mystique, Here’s The Upshot
|Interpretable A.I.||Essential for business confidence and in critical decisions is the ability
not just to provide accuracy with Machine Learning but also the why and how.
In short, interpretability means to determine a representation in terms of human understanding
of the results; with few parameters (i.e. linear regression) this is straightforward.
At the other end, Deep Neural Networks (DNN) are effective in finding
subtle relationships among many features but are hard to interpret.
Recently developed methods to analyze DNN include LIME
(Local Interpretable Model-Agnostic Explanations)
and DeepLIFT (Deep Learning Important Features)
With using alternatives to DNN, the common belief is that interpretability goes
at the expense of accuracy, on which assertion some disagree with.
Decoding the Black Box
Guide to Interpretable Machine Learning
|The use of Neural Networks||An exciting area in the deep learning space is the use of neural networks for solving PDEs, equations which dictate the majority of geophysical phenomena. Through tailoring of the cost function, physics-informed neural networks (PINNs) have recently been shown to accurately solve a variety of PDEs. Early attempts in geophysics have been published for solving both the Eikonal and the wave equation. Whilst it is still unclear whether PINNs will reach the precision of our waveform modeling procedures, they are likely to be a fierce competitor with respect to compute time.
The underlying principles of PINNs are detailed in this page.
An example of such a network being to solve the wave equation is illustrated by this paper.
And, for those ready to get your hands dirty, checkout the DeepXDE python library.
|Quick and easy A.I.||Its' great to try simple examples to see what A.I. can do.
There are multiple sites where you can try examples for free and see the results.
In the examples below you can upload images and see how ML systems perform classifications
and extractions and what data they return.
Google – Image classification
Microsoft – Image classification
When you want to step into run your own more domain specific data (e.g. timeseries or
multiple attribute data) then many of the ‘AI Platforms’, like Dataiku and DataRobot allow
you to register and run free versions. These systems can run ‘code free’, so if you can use
Excel then should be able to run those. These are great ways to explore quickly what A.I. can do
and see if it might be relevant for you and your data challenges!
|Explainable A.I.||Explainable Artificial Intelligence (XAI) tries to open the black box of Machine Learning models such that their behavior can be understood by humans. Google Cloud's A.I. Explanations provide a set of tools and frameworks to explain how much each feature in your model contributed to the predicted results for classification and regression tasks. More specifically, SHAP (SHapley Additive exPlanations) is a popular XAI tool based on a solution in cooperative game theory. It can explain the output of any machine learning model with rich visualizations that are friendly for end users.|
|A.I. Challenges||Historically, the Imagenet Challenge has allowed researchers to develop ground-breaking machine learning methods on open data, enabling reproducible, comparable progress in computer vision.
In geoscience, efforts such as the SEG contest on facies prediction have inspired geoscientists to engage in the field of AI and serve as an excellent entry point for machine learning in geoscience.
Currently ongoing, the FORCE machine learning contest on wells and seismic provides a labeled dataset for facies prediction from wireline logs and a seismic dataset for fault detection.
These and other collaborative challenges will help to inspire future geoscientists and breakthrough technologies in applied machine learning for geoscience.
|Deep Learning for A.I. (2)||There is plenty of online training material on Deep Learning.
This week we recommend three sources that are very useful for illustrating the practicalities of Deep Learning. They are real fun to use!
TensorFlow playground (already discussed in a different context) provides simple two-dimensional examples of feed-forward neural networks, mostly for classification, and displays the results in a very useful way for somebody who is new to neural networks.
3D Visualization of a Convolutional Neural Network shows the details of the structure and performance of a simple convolutional neural network applied to the classical MNIST dataset.
GAN Lab explains Generative Adversarial Networks, and it really helps understand the interaction between the Generator and the Discriminator.
|U-net||Understanding what happens in images in crucial in the field of machine vision. This problem is broken up into separate but similar topics, such as classification, localization, object detection, semantic segmentation and image segmentation. Without realizing it, geoscientists face similar challenges. Think first break picking or salt interpretation. One of the workhorses for image segmentation problems is the U-net and to get ahead in the field, or simply grasp what your colleagues have developed for you now, one should really
have a basic understanding of this algorithm. Here are three useful links:
- Convolutional Networks for Biomedical Image Segmentation (video) - Beginner
- Convolutional Networks for Biomedical Image Segmentation (paper) - Intermediate
- U-net application fo TGS challenge - Advanced
|Deep Learning for A.I. (1)||Drastic improvements in hardware performance (GPU) enabled wide spread use of Deep Neural Networks (DNN). Combined with the Convolutional Neural Network (CNN) approach, they complement seismic workflows very well; fault detection, time lapse, inversion seismic-log integration, etc.
In application, careful consideration is advised: non transparency (black box), dependence on training data, outcomes being approximations, sometimes artefacts.
However, because of the multilayered architecture, Deep Learning has proven ‘unreasonably effective’, and improved understanding through research (MIT) will enable novel breakthroughs.
- TensorFlow Neural Network (Beginner)
- Deep Learning Specialization (Intermediate)
- Seismic Deep Learning libraries (Advanced)
|Hands-on A.I. exercises||A hurdle for many wanting to gain hands-on experience with AI is setting up a development environment - hours of frustration trying to install Python on Windows, we have all been there! Google's CoLab provides an online Jupyter-like environment with FREE GPU resources where you can experiment to your heart's content. Whilst Google has already provided a number of data science tutorials, one of the great benefits of CoLab is it is possible to open any .ipynb file. Whether you are looking at csv files, images or jumping right into manipulating segy data, there are hundreds of geoscience-specific examples sitting in open GitHub repositories. Here are three notebooks to get you started:
- Analysing thin section compositions (Beginner)
- An image segmentation example from the TGS salt detection Kaggle competition (Intermediate)
- Seismic inversion on the Volve dataset (Advanced)
|A.I. Give it a go - it won't bite||If you haven’t had a go before – try it, get your hands (digitally) dirty. You can play without breaking anything (or in some cases even without installing anything). It can help you understand what’s possible. It can help at work, in your AI studies or across the rest of your life. Thankfully these days you don’t have to be a coding supremo to take those first steps. Much of the AI world is moving towards ‘low-code’ or even ‘no-code’, so you can do some pretty impressive AI stuff without leaving the comfort of a friendly app, whether that be on your phone or laptop. Below are a few cool places where you can start exploring the art-of-the-possible – hopefully it will inspire you and be fun, enjoy!
- AI Experiments with Google (Beginner)
- Machine Learning Experiments with GitHub (Beginner to Intermediate)
- Anaconda, incl. Orange (no code), Jupyter, Spyder (Python) & RStudio (low to high code) (Beginner to Advanced)
|The pandemic and A.I.||The coronavirus outbreak put us in unprecedented times. This week we take a special look at the role A.I. can play in battling the pandemic as well as transforming the healthcare practice. Check out the latest issue of Nature Machine Intelligence for a general read about the potential advantages and challenges of deploying A.I. in the pandemic. With no prior medical expertise required "A.I. for Medicine Specialization" teaches how to apply A.I. tools to medical diagnosis, prognosis and treatment, including working with 2D and 3D medical image data. You can obtain open dataset, share code and models, and enter competitions on the largest machine learning community Kaggle to join the battle against Covid-19 as an A.I. practitioner.
- A path for A.I. in the pandemic (Beginner)
- AI for Medicine Specialization (Intermediate)
- Kaggle ML Community (Advanced)
|Staying up-to-date with A.I.||This week's focus is on staying up-to-date with the rapidly moving field of AI.
"Two-minute papers" is a video podcast series that aims to distill the hottest and most fascinating research in the field of computer vision and machine learning into a format accessible for everyone. The artificial intelligence podcast by Lex Friedman is an excellent resource for interviews with researchers around the field of AI and machine learning. Arxiv-sanity preserver is your one-stop-shop to the world of AI and machine learning preprints, where the most recent publications from the ArXiv can be found in one place.
- Two-minute Papers (Beginner)
- The Artificial Intelligence Podcast (Intermediate)
- ArXiv Sanity Preserver (Advanced)
|Learning A.I.||To get started, the focus naturally falls on learning. If you have been looking for the right entry, here are three (free) courses for you to consider:
- AI For Everyone (Beginner)
- Practical Deep Learning for Coders, v3 (Intermediate)
- UVA Deep Learning Course (Advanced)
|From Recurrent Neural Networks to Markov Chains||Karpathy’s blog provide a fun introduction to Recurrent Neural Networks (RNNs) through the character-by-character generation of text,
Johnson, in his Stanford University course, discusses more in depth the theory and applications of RNNs and LSTMs (Long Short-Term Memory),
You might say: what is in it for a geoscientist? Well, vertical sequences of geological facies can be regarded as a sequence of characters! Generating some synthetic sequences after training an RNN on vertical geological logs is an interesting idea proposed by Talarico, Leao and Grana in their paper “Comparison of Recursive Neural Network and Markov Chain Models in Facies Inversion”, available on EarthDoc.
Questions? Ideas? Contact us!