As we witness information growth driven by recent developments of the Internet of Things (IoT), Complex Event Processing (CEP) comes in, seeking to make sense of an otherwise unintelligible sea of data.
Enormous amounts of events are produced in real time from our surroundings: cell phones, wearables, TV setup-boxes, CCTV Cameras... Uses for the information patterns that can be extracted from raw events are still to be found. Some may remember the use of Jawbone sleep tracking information during an earthquake to establish the worst affected areas (where most people woke up), combining geo-location with biometric sensor information.
David Luckham coined Complex Event Processing (CEP) expression in his 2002 book "The Power of Events", as a technique for processing real time events and extracting meaningful information from event streams as they arrive. The focus here is the transformation from data to information, into knowledge.
CEP can be seen as a category of Business Intelligence (BI), since both help decision making by attempting to provide the right information. CEP real time scope formulates very different use cases; one cannot expect to detect and act upon online transaction fraud, on the monthly BI report.
Rule-based technology is the basis for CEP, linking unrelated raw events. The traditional rule logic can be achieved by IF – THEN statements, once the raw events are identified. However nesting these statements is not the best way for evaluating and reusing test sequences, even if the rules are prioritized:
1) IF client account > 200 credits THEN status = Silver
2) IF status is Silver THEN unlock promotion 25%
3) IF client account has > 500 credits THEN status = Gold
4) IF status is Gold THEN unlock promotion 35%
5) IF client buys product THEN award = 50 credits
Applying forward chaining (IF A =>B and B=>C THEN A =>C) we can infer:
If client has > 500 credits then the promotion is 35%, since the bonus is Gold.
Rule #5 (sum 50 credits) can be a problem since it implies re-evaluation of all the rules. The approach can be extended with a state machine improving rule re-evaluation. Rules will trigger state transitions until reaching a final state or a timeout.
Even with state machines the quantity of rules will eventually become a problem due to the real-time constraint. That’s where the RETE Algorithm comes in, developed by Dr Charles L. Forgy on the 70’s. The break-through was to create a network of nodes aggregating objects that satisfy associated conditions. In our example, we can create two classes of objects (Account and Loyalty Card), and create a tree with all the conditions for each; afterward connect both trees creating nodes where the union of the conditions will evaluate into a final state (in this case a promotion). The main emphasis is that rule evaluation is done by traversing the tree network, as opposed to sequential evaluation. (If interested check this link)
David Luckham coined Complex Event Processing (CEP) expression in his 2002 book "The Power of Events", as a technique for processing real time events and extracting meaningful information from event streams as they arrive. The focus here is the transformation from data to information, into knowledge.
CEP can be seen as a category of Business Intelligence (BI), since both help decision making by attempting to provide the right information. CEP real time scope formulates very different use cases; one cannot expect to detect and act upon online transaction fraud, on the monthly BI report.
Rule-based technology is the basis for CEP, linking unrelated raw events. The traditional rule logic can be achieved by IF – THEN statements, once the raw events are identified. However nesting these statements is not the best way for evaluating and reusing test sequences, even if the rules are prioritized:
1) IF client account > 200 credits THEN status = Silver
2) IF status is Silver THEN unlock promotion 25%
3) IF client account has > 500 credits THEN status = Gold
4) IF status is Gold THEN unlock promotion 35%
5) IF client buys product THEN award = 50 credits
Applying forward chaining (IF A =>B and B=>C THEN A =>C) we can infer:
If client has > 500 credits then the promotion is 35%, since the bonus is Gold.
Rule #5 (sum 50 credits) can be a problem since it implies re-evaluation of all the rules. The approach can be extended with a state machine improving rule re-evaluation. Rules will trigger state transitions until reaching a final state or a timeout.
Even with state machines the quantity of rules will eventually become a problem due to the real-time constraint. That’s where the RETE Algorithm comes in, developed by Dr Charles L. Forgy on the 70’s. The break-through was to create a network of nodes aggregating objects that satisfy associated conditions. In our example, we can create two classes of objects (Account and Loyalty Card), and create a tree with all the conditions for each; afterward connect both trees creating nodes where the union of the conditions will evaluate into a final state (in this case a promotion). The main emphasis is that rule evaluation is done by traversing the tree network, as opposed to sequential evaluation. (If interested check this link)
But the greatest challenge presented today by IoT to CEP is:
The nature of IoT events is not certain and subject to faults
What do (RFID, smart watches with bio sensors, GPS, WIFI, Bluetooth…) have in commun?
The backbone of IoT is prone to: signal disturbance, sensor miscalibration, tamper, energy blackouts…
Future application of CEP is assimilating many of the ideas behind (AI) Expert Systems developed on the past, as they share the need for inference engines to predict future behavior:
- Rules will be called into question. Systems must be able to work with incomplete data; each rule has to deal with uncertain knowledge and confidence factors. Known as ‘fuzzy logic’.
- Prediction analytics based on dynamic Bayesian models, use historical data in order to extrapolate future occurrences statistically.
The CEP field is growing fast with the advent of IoT, the tools are in place to harvest the information and produce the so desired knowledge!
The nature of IoT events is not certain and subject to faults
What do (RFID, smart watches with bio sensors, GPS, WIFI, Bluetooth…) have in commun?
The backbone of IoT is prone to: signal disturbance, sensor miscalibration, tamper, energy blackouts…
Future application of CEP is assimilating many of the ideas behind (AI) Expert Systems developed on the past, as they share the need for inference engines to predict future behavior:
- Rules will be called into question. Systems must be able to work with incomplete data; each rule has to deal with uncertain knowledge and confidence factors. Known as ‘fuzzy logic’.
- Prediction analytics based on dynamic Bayesian models, use historical data in order to extrapolate future occurrences statistically.
The CEP field is growing fast with the advent of IoT, the tools are in place to harvest the information and produce the so desired knowledge!