Machine learning at credit card providers: the Amex story

More and more financial services companies use machine learning to store, extract and evaluate data. The credit card industry is one sector where this is happening, with one of the major names, American Express (Amex), harnessing machine learning techniques to great success.

Credit card providers use machine learning techniques to attract new customers, drive up turnover, offer personalized recommendations to users and identify fraudulent operations. Chao Yuan, Senior Deputy Chairman of American Express, discussed at an event in Silicon Valley organized by a think tank how the company is harnessing machine learning with two aims:to make decisions to support its market strategy and make optimal use of its data.

1. Attracting new customers

Machine learning allows companies to tailor products to consumer needs and exploit every opportunity for multi-channel customer acquisition initiatives. Prior to bulk use of data and machine learning, new client acquisition at Amex was nearly entirely via promotions sent by direct mail to consumers.

Now, thanks to the internet and machine learning, any company can create predictive models that serve of make customized offers to potential clients. American Express has gone from attracting new clients exclusively by mail to attracting plenty of new business online. This is good news for the company: mailing costs are higher than those associated with digital channels.

2. Better data, more transactions

American Express' business can be summarized in a single phrase: more transactions, greater profits; fewer transactions, less profit. Amex currently has more than 100 million cards in circulation worldwide, handling close to 950 billion transactions between individuals and businesses. In 2014 Amex earned nearly 5.9 billion dollars, 4% more than in 2013.

What does American Express do to drive up its profits each year? It harnesses data analysis techniques to extract valuable information from all possible combinations in transactions between buyers and sellers using their cards. Amex charges a percentage of each of these transactions. By driving up transaction numbers, it also lifts its revenue.

Understanding how and why consumers use their cards in relation to sellers' products and services makes it easier to ramp up transaction numbers.

To process large data volumes American Express has implemented its current Sync Platform in Apache Hadoop, which works with MapReduce and a Distributed File System (HDFS). This system replaced the previous analysis techniques using relational databases. Some of the key characteristics of the platform are as follows:

- It was set up three years ago as a ten-node Hadoop cluster. The true volume of the infrastructure now is unknown.

- Generally, each of the nodes has eight x86 cores or more per base and the entire infrastructure should have some 300 cores in total.

- Each rack (combination of Hadoop nodes) has nearly a petabyte of data.

- Each machine has two 10 Gb Ethernet networks to connect the cluster nodes.

- It runs both in real time and batch applications.

3. Tailored recommendations

American Express uses machine learning techniques in mobile applications. Data usage has jumped on such devices, which has generated much of the demand. At present Amex uses algorithms to shape personalized recommendations for products and services, such as recommending restaurants from its list of associated businesses.

Provided that the user has given permission for it to do so, the algorithm compiles all available information on consumer habits, income and spending and thus generates a predictive model and recommendations that suit their expenditure and needs. Amex is able to recommend restaurants, with strong probabilities of success, where a user will be able to buy its partners' products while paying with one of its cards, with benefits offered to the client for doing so.

Compiling such commercial and financial information requires powerful and fast infrastructure. So does model creation and data analysis. The Apache Hadoop platform deployed by Amex is up to the task.

4. Detecting fraudulent transactions

The American Express machine learning and data analysis infrastructure allows fraudulent transactions with one of their credit cards to be identified in milliseconds. Of course, such early fraud detection would make no sense if each card payment had to be delayed for hours or even minutes.

What significant progress has been made in Hadoop as MapR? The ability to make real time decisions and evaluate all data, not just a sample, without any interim steps and without affecting users. MapR allows Amex to generate far more precise prediction patterns, because they are generated using all past transaction data and not just a small sample.

This has three important benefits:

- The number of legal transactions that are denied because they are mistakenly thought to be fraudulent is reduced considerably (these are known as false positives). That aim is to detect illegal operations without generating false positives and thus annoying users.

- Increased detection of unprecedented irregular transactions.

- MapR supports a number of supervised and non-supervised learning methods, making for a more efficient predictive model that can updated in less time. This means lower costs for companies.

Follow us on @CIBBVA