Botnet Detection Using On-line Clustering with Pursuit Reinforcement Competitive Learning (PRCL)

Yesta Medya Mahardhika, Amang Sudarsono, Ali Ridho Barakbah


Botnet is a malicious software that often occurs at this time, and can perform malicious activities, such as DDoS, spamming, phishing, keylogging, clickfraud, steal personal information and important data. Botnets can replicate themselves without user consent. Several systems of botnet detection has been done by using classification methods. Classification methods have high precision, but it needs more effort to determine appropiate classification model. In this paper, we propose reinforced  approach to detect botnet with On-line Clustering using Reinforcement Learning. Reinforcement Learning involving interaction with the environment and became new paradigm in machine learning. The reinforcement learning will be implemented with some rule detection, because botnet ISCX dataset is categorized as unbalanced dataset which have high range of each number of class. Therefore we implemented Reinforcement Learning to Detect Botnet using Pursuit Reinforcement Competitive Learning (PRCL) with additional rule detection which has reward and punisment rules to achieve the solution. Based on the experimental result, PRCL can detect botnet in real time with high  accuracy (100% for Neris, 99.9% for Rbot, 78% for SMTP_Spam, 80.9% for Nsis, 80.7% for Virut, and 96.0% for Zeus) and fast processing time up to 176 ms. Meanwhile the step of CPU and memory usage which are 78 % and 4.3 GB  for pre-processing, 34% and 3.18 GB for online clustering with PRCL, and  23% and 3.11 GB evaluation. The proposed method is one solution for network administrators to detect botnet which has unpredictable behavior in network traffic.


Botnet Detection; Maliciouse Software; On-line Clustering; Pursuit Reinforcement Competitive Learning


S. Saad et al., “Detecting P2P Botnets through Network Behavior Analysis and Machine Learning,” Ninth Annu. Int. Conf. Privacy, Secur. Trust, pp. 174 – 180, 2011.

C. Chen and H. Lin, “Detecting botnet by anomalous traffic,” J. Inf. Secur. Appl., vol. 21, pp. 42–51, Apr. 2015.

D. Garant and Wei Lu, “Mining Botnet Behaviors on the Large-Scale Web Application Community,” in 2013 27th International Conference on Advanced Information Networking and Applications Workshops, 2013, pp. 185–190.

W. T. Strayer, D. Lapsely, R. Walsh, and C. Livadas, “Botnet Detection Based on Network Behavior,” in Botnet Detection, vol. 36, no. August, Boston, MA: Springer US, 2008, pp. 1–24.

E. B. Beigi, H. H. Jazi, N. Stakhanova, and A. A. Ghorbani, “Towards effective feature selection in machine learning-based botnet detection approaches,” in 2014 IEEE Conference on Communications and Network Security, CNS 2014, 2014, pp. 247–255.

D. Zhao, I. Traore, B. Sayed, W. Lu, S. Saad, A. Ghorbani, and D. Garant, “Botnet detection based on traffic behavior analysis and flow intervals,” Comput. Secur., vol. 39, pp. 2–16, 2013.

A. Shiravi, H. Shiravi, M. Tavallaee, and A. A. Ghorbani, “Toward developing a systematic approach to generate benchmark datasets for intrusion detection,” Comput. Secur., vol. 31, no. 3, pp. 357–374, May 2012.

S. García, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” Comput. Secur., vol. 45, pp. 100–123, Sep. 2014.

A. J. Aviv, “Challenges in Experimenting with Botnet Detection Systems,” USENIX 4th CSET Work. San Fr. CA, pp. 1–8, 2011.

F. V. Alejandre and N. C. Cort, “Botnet Detection using Clustering Algorithms,” vol. 118, pp. 65–75, 2016.

K. Huseynov, K. Kim, and P. D. Yoo, “Semi-supervised Botnet Detection Using Ant Colony Clustering,” vol. 31, no. The 31th Symposium on Chryptography and Information Security Kagoshima, pp. 1–7, 2014.

D. Zhao, I. Traore, A. Ghorbani, B. Sayed, S. Saad, and W. Lu, “Peer to Peer Botnet Detection Based on Flow Intervals,” Inf. Secur. Priv. Res., vol. 3, no. 1, pp. 87–102, 2012.

G. Kirubavathi and R. Anitha, “Botnet detection via mining of traffic flow characteristics R,” Comput. Electr. Eng., vol. 50, pp. 91–101, 2016.

S. Miller and C. Busby-earle, “The Impact of Different Botnet Flow Feature Subsets on Prediction Accuracy Using Supervised and Unsupervised Learning Methods,” vol. 5, no. 2, pp. 474–485, 2016.

I. Y. P. Tiyas, A. Barakbah, T. Harsono, and A. Sudarsono, “Intrusion Detection with On-line Clustering Using Reinforcement Learning,” in Proceeding The Third Indonesian-Japanese Conference on Knowledge Creation and Intelligent Computing, 2014, pp. 30–37.

A. Barakbah, “Special Issues on Clustering,” Knowledge Engineering Research Group PENS, Ed. EEPIS, 2016, pp. 1–80.

A. Barakbah and K. Arai, “Pursuit Reinforcement Competitive Learning,” in The 2nd International Seminar on Information and Communication Technology Seminar (ICTS), 2006.

DOI: 10.24003/emitter.v6i1.207


  • There are currently no refbacks.

Copyright (c) 2018 EMITTER International Journal of Engineering Technology

EMITTER Journal Editorial Office


Politeknik Elektronika Negeri Surabaya

Jl. Raya ITS - Kampus PENS Sukolilo Surabaya 60111, INDONESIA   Telp : +62 31 594 7280   Fax : +62 31 594 6114