Neuro Symbolic Learning

In the area of hybrid machine learning, the focus is on combining inductive learning techniques and analytical learning techniques. Neural Networks are very successful in real-time applications, because they are robust in the presence of noisy data, and all what they need to learn is data, which in many real-time domains is easy to obtain. At the same time knowledge based systems rely mostly on explicitly stated if-then rules describing the domain over which they reason. Using sound and complete logical inference rules, the first order logic is a powerful tool for reasoning about knowledge.

At the same time both the above techniques have their disadvantages as well. Neural Networks have a very strong statistical bias, and the trained network is usually very hard to understand. This in turn means that one can't reason about the results obtained through such a network on a logical level. As an example why should a real-time safety critical system rely on the performance of a network when all what it knows is the matrix of real numbers valued weights on the links of the units connected in a feed forward neural network?

Knowledge based systems which rely exclusively on explicitly stated if-then rules, are no longer powerful when the rules that describe the domain could be wrong, or one missing rule could mean an absurd inference. This means that those systems which rely on the logical inference rules can no longer be considered safe to reason about incomplete or even wrong domain representations.

One needs to have a trade-off between the knowledge and the data, and this is one of the motivations for combining purely inductive learning schemes like neural network and purely logic based knowledge systems like first order logic.

Research within the hybrid paradigm has reached a certain level of maturity in the sense that we have some good hybrid learning schemes which combine the 3 layer feed forward neural network and propositional logic programming.

For a detailed review of this, the following surveys are available:

Learning first-order rules in a neural network

In my Masters thesis work, I investigated the problem of combining the first-order logic knowledge representation scheme with neural networks. The basic approach remained the same as proposed by Jude Shavlik and Geoff Towell in their KBANN system.

The main problem in tackling this arose from the representational issues for first-order logic rules within a neural network, and learning those rules. Lokendra Shastri's SHRUTI system provided some clues as to how a first order logic could be encoded in a neural network. However nothing very specific was mentioned on how to do learning in those networks. The reason why learning became difficult with SHRUTI type networks was mainly because of the way the rules were encoded in the neural network.

We departed from SHRUTI's approach in quite a radical way. We proposed a new encoding scheme for first-order logic rules, although we have to impose a couple of restrictions and indeed our system does not handle several kinds of rules which SHRUTI handles at present. However, once we have encoded the rules, we can use a non-supervised hebbian type learning for learning the rules. The novelty of our approach lies in using the powerful spike-based neural networks in contrast to the conventional static feed-forward sigmoidal unit networks. The first experiments conducted on this system show promise that the system can be improved further on for dealing with a fairly larger class of first order logic and at the same time implement learning in the recursive part of the network.

For an exhaustive review of this approach, you may have a look at my Master thesis.

Some useful links related to research in rule extraction.

Other related links



Valid HTML 4.01!

Last updated by Ashish Darbari on Nov 11, 2002