Dynamic feature selection for clustering high dimensional data streams
Change in a data stream can occur at the concept level and at the feature level. Change at the feature level can occur if new, additional features appear in the stream or if the importance and relevance of a feature changes as the stream progresses. This type of change has not received as much attention as concept-level change. Furthermore, a lot of the methods proposed for clustering streams (density-based, graph-based, and grid-based) rely on some form of distance as a similarity metric and this is problematic in high-dimensional data where the curse of dimensionality renders distance measurements and any concept of “density” difficult. To address these two challenges we propose combining them and framing the problem as a feature selection problem, specifically a dynamic feature selection problem. We propose a dynamic feature mask for clustering high dimensional data streams. Redundant features are masked and clustering is performed along unmasked, relevant features. If a feature's perceived importance changes, the mask is updated accordingly; previously unimportant features are unmasked and features which lose relevance become masked. The proposed method is algorithm-independent and can be used with any of the existing density-based clustering algorithms which typically do not have a mechanism for dealing with feature drift and struggle with high-dimensional data. We evaluate the proposed method on four density-based clustering algorithms across four high-dimensional streams; two text streams and two image streams. In each case, the proposed dynamic feature mask improves clustering performance and reduces the processing time required by the underlying algorithm. Furthermore, change at the feature level can be observed and tracked.
open access article
Citation : Fahy, C. and Yang, S. (2019) Dynamic feature selection for clustering high dimensional data streams. IEEE Access, 7(1), 127128-127140.
Research Institute : Institute of Artificial Intelligence (IAI)
Peer Reviewed : Yes
Showing items related by title, author, creator and subject.
The use of Additive Manufacture (AM) in New Product Development (NPD) has increased over recent years. AM enables greater creativity in design, reduced tooling cost and faster product development time. However, there is a ...
Hill, Lisa (Thesis or dissertation / Masters / MPhil)Hooded garments are a widespread fashion article and readily lend themselves as a disguise in terms of covering the hair and majority of the face. The presence of external features and maintaining the context in which a ...
Alazemi, Fayez (Thesis or dissertation / Doctoral / PhD)The gap between business goals (problem domain), such as cost reduction, new business processes, increasing competitive advantage, etc., and the supporting Information Technology infrastructure (solution domain), such as ...