"This paper describes the design, implementation, and evaluation of WASp, a tool that automatically reverse-engineers proprietary wireless protocols that run over IEEE 802.15.4. Such protocols are commonly used in IoT systems, and two are studied in this paper: a smart plug system and a platform screen door system. Such protocols often depend on obscurity, because they 1) use proprietary binary encoding for the packets, 2) lack documentation, and 3) do not necessarily use encryption. Therefore it is a worthwhile goal to assess their security through reverse-engineering. The proposed WASp tool takes as input captured packets (pcap files) and some context information (e.g. number of nodes involved in the communication) and aims to first, reconstruct the packet format of the unknown protocol and then second, generate ("spoof") packets that conform to the protocol.
The reviewers appreciated practical problem studied by the paper, noting that it will become more important as IoT deployments increase. The implementation and practical testing of the tool adds value and shows that the protocols studied can be reverse-engineered to some extent. The paper is in general well-written and easy to understand.
The main concern shared by all reviewers is that the evaluation of WASp does not use ground truth (i.e. reverse-engineer known protocols and evaluate WASp by comparing its output to the known packet format). The evaluation metrics used are the "spoofing success rate" (a spoofed packet is considered a success, if any response packet is detected for it) and the "entropy reduction" (this is not formally defined in the paper, but it relates to the reduction in the number of packets that the attacker - before and after running WASp - considers as potentially valid protocol packets). However, due to legal and safety reasons, the "spoofing success rate" cannot always be used, which was the case in the second system studied (platform screen door). Also, the entropy reduction method would benefit from a clearer definition and more justification as an evaluation metric. Overall, wider and more rigorous evaluation is needed, and this is a useful direction for future research.