EVALUATION OF EARLY SEASON MAPPING OF INTEGRATED CROP LIVESTOCK SYSTEMS USING SENTINEL-2 DATA

Various approaches were developed considering the need to increase agricultural productivity in cultivated areas without more deforestation, such as the Integrated Crop livestock systems (ICLS). The ICLS could be composed of annual crops followed by pastureland with the presence of cattle. Due to the high temporal dynamic of rotation between crops over the season, monitoring these areas is a big challenge. Also, agricultural organizations worldwide highlight the need for early-season maps for this kind of work. In this context, this study evaluated the potential of open data (Sentinel-2) data to map ICLS areas. The performance of two classifiers was evaluated: one of Machine Learning (random forest) and the other of Deep Learning (LSTM). Three different time windows of data were tested (Entire season, 180 days, and 120 days). Using the RF classifier, it was possible to achieve satisfactory results (Overall accuracy higher than 80%) for the early season (180 days). However, further studies are needed to explain better the lower(when compared to Random Forest) accuracy achieved by LSTM net (0.79 % for 180 days) and compare the results achieved here with results for a study area with different rates of cloud cover.


INTRODUCTION
In recent years, there has been an increasing search for regenerative agricultural practices worldwide because traditional agriculture is not able to keep up with the population growth rate in a sustainable way anymore (Ray et al., 2019). In this circumstance, aiming at regenerative agricultural practices, one studied approach is the Integrated Crop Livestock System (ICLS), where in the same field there is a synergism between crop and livestock, increasing the productivity without needing more land (Cordeiro et al., 2020). This practice could significantly improve food production regarding sustainability and productivity since it brings nutrients for degraded soils, improving the livestock system at the same time that the area could be used for crop production (Gil et al., 2018).
The ICLS has three main objectives, (i) reduce the soil cyclical nutrients loss and consequently increase plant productivity, (ii) organize agricultural practices in a way that contribute to ecosystem services, and (iii) increase economic resilience to adverse hypotheses from an economic and environmental point of view (Moraine et al., 2014). For these reasons, some countries, such as Australia, Brazil, France, New Zealand, and the United States, are stimulating the adoption of ICLS in agricultural areas (Garrett et al., 2017).
Thus, the Brazilian government has a plan for mitigation and adaptation to climate change, where one of the goals is to expand from 17 to 35 Million hectares of ICLS areas (EMBRAPA, 2021). However, the main challenge in remotely identifying and monitoring this type of system is the high complexity dynamic resulting from the succession of different land cover and management.
In this framework, agricultural mapping using satellite imagery increases significantly with the growing availability of high temporal, spatial, and spectral data (Weiss et al., 2020). Optical There are a few studies published regarding the use of remote sensing data to map ICLS (Kuchler et al., 2020, Almeida et al., 2021. Further, due to the high complexity of the time series, several studies usually use a time window composed by the entire season as input (Tian et al., 2021). This limits the application of methodologies for in-season purposes as well as consumes more time and resources to be done . Recently, in the annual report of the European Commission for agricultural monitoring, the early-season mapping is highlighted as one of the three main needed improvements for agricultural monitoring (Charvat et al., 2020).Some authors are working on using Sentinel-2 data to provide early-season mapping of crops, achieving high accuracy for some crop types (Nasrallah et al., 2018, Tian et al., 2021 In this context, deep learning algorithms represent state-of-the-art for crop type mapping, which can deal with highly complex information as well as tend to perform better with reduced time series (Aduvukha et al., 2021).
Among the deep learning algorithms, deep neural networks are the most used for remote sensing image analysis (Ma et al., 2019), with emphasis on Recurrent Neural Network -RNN, for problems regarding the temporal dimension (Zhong et al., 2019). The RNNs, are capable of dealing with data sequences in such a way that the output of the previous time-step is the input data for the current step (Campos-Taberner et al., 2019), handling temporal problems (Ma et al., 2019). Regarding the RNNs, the Long-Short-Term-Memory(LSTM) is a complex Recurrent Neural Network currently considered the state-of-the-art in terms of dealing with complex problems in the temporal dimension (Almeida et al., 2021). If compared with a traditional Recurrent Neural Network (RNN), the main idea behind LSTMs is an adapted cell tends to detect long-term dependencies between variables, besides the fact that it converges faster than the usual RNNs (Géron, 2019). On the other hand, it is also essential to always test ro-The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France bust and more easy-to-implement existing classifiers to find the most feasible approach for practical implementation. Within this framework, the Random Forest classifier could be considered one of the most robust and used machine learning algorithms for classification (Belgiu and Drȃguţ, 2016). Furthermore, it is essential to compare the result obtained by different classifiers with ultimately approaches to perform the classification.
In this background, the present study aims to (i) evaluate the potential of open remote sensing data to map integrated croplivestock systems, (ii) evaluate the possibility of performing an early-season classification for those systems, and (iii) compare the performance of a state-of-the-art deep learning algorithm (LSTM) with a robust machine learning classifier (Random Forest) for classifying ICLS areas.

STUDY AREA AND DATA
The study area comprises a farm located in the western portion of São Paulo state, and its surrounding fields in the municipality of Caiuá, at coordinates 21°36'26.3"S and 51°51'57.9"W, with an area equivalent to 83 squared kilometers ( Figure 1). According to the Koppen classification, the region has a climate Aw type, corresponding to tropical climate conditions with the dry season in the winter (Alvares et al., 2013). The average temperature of the region is 24.1ºC, and the average annual precipitation is 1496 mm (considering the period from 2013 to 2018). At the Campina farm, the process of implementing the ICLS started in 2013 (Cordeiro et al., 2020). In this ICLS, soybean and forage species (Brachiaria and Panicum) are inter-cropped in the first and second seasons. At the first season (Summer) there are two soybean cycles followed by mixed pasture with livestock presence in the second season (Winter).

Ground truth data
2955 ground reference points of eight different land cover classes were collected for this study area. The labels of each point were attributed to segments previously generated,using the watershed algorithm, based on temporal compositions of Enhanced Vegetation Index (EVI) done using Sentinel-2 imagery (Hossain and Chen, 2019).

Satellite data
Images from the Sentinel-2 satellites freely available at the European Space Agency (ESA) Hub were acquired from September 2019 to August 2020. The pre-processing of images from both satellites was carried out by the Sentinel Application Platform (SNAP) software offered by the ESA. Regarding the pre-processing of products from the Sentinel-2 satellites, geometric, atmospheric, and radiometric corrections were carried out, as proposed by (Ranghetti et al., 2020). In total, 74 cloud-free Sentinel-2 images(L2A) were obtained.

Early season
Aiming to provide early season classification, three different datasets were generated to input in the classifiers. The first one comprised the entire season (from September to August), the second one was composed of 180 days of data (from January to June). Finally, the last one was composed of 120 days of data (from January to April). January was chosen as the beginning of smaller datasets because, for ICLS areas, the peak of the annual crop usually occurs in January, and the pasture starts to be implemented in April (Dos Reis et al., 2020). Thus, the hypothesis is that those short time windows would be the most representative since both comprise the annual crop and the beginning of the pasture season. Selecting those time windows, is expected to overcome the main difficulty related with early crop mapping, which is the lack of meaningful temporal information (Kwak et al., 2021).

Classifiers
Considering its robustness and its capacity to deal with outliers, low processing costs, and flexibility (Belgiu and Drȃguţ, 2016) the Random Forest classifier was tested in the present study. Random Fores is not known by recognition of temporal dependencies a sit is probably needed in this study but, many studies already highlighted its potential to select appropriate features for classification, especially when dealing with multispectral data as Sentinel-2 (Benevides et al., 2021). This classifier is considered well established and broadly applied for crop identification based on satellite data (Saini and Ghosh, 2018).
Also, considering the high temporal dynamic of the target crop in this study (ICLS), it could be supposed that the LSTM net would be adequate to identify the temporal pattern of the ICLS area.
Since once the net could be able to "remember" that the spectral pattern of an annual crop occurred before a pasture spectral pattern and further classify it as an ICLS area.
Thus, both classifiers were tested using the three different time windows. Classifications were conducted in a python environment, mainly using sci-kit-learn and Tensorflow (Géron, 2019). The classifiers performance were evaluated based on the overall accuracy, f-score, and confusion matrices.For both classifiers the proportion train, evaluation and test was 70%,20% and, The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France 10%(used for prediction). Also, the Feature importance score was calculated for RF classifications to identify the relevant dates and indices for the classification.

Prediction
Finally, the best result for each classifier was used to predict the land cover on the study area and surrounding fields, many of them were the test set (which was not used as input in the training). This is expected to better provide an overview of the potential of the model to predict for unknown fields. Thus the spatial distribution and area of each class could be observed.

Exploratory analysis and dataset preparation
Concerning the dataset preparation and exploratory analysis, it was observed that the classes were highly unbalanced, being the class 'Cultivated Pasture' representing more than 50% of the samples. Also, the classes Wet Areas and Water were joined due to the few samples of each class and the impossibility of distinguishing among them relying on ground truth data. Also, to deal with the problem of unbalanced data, the SMOTE algorithm was performed in the training dataset (Wang et al., 2006). This algorithm generates synthetic data for classes with few samples until they achieve the same number of samples as the predominant class (Cultivated Pasture) (Wang et al., 2006).

Early Season
Regarding the early season results, the random forest classifier using the Sentinel-2 data achieved superior performance in terms of overall accuracy even using a reduced time window (Table 2). The overall accuracy was even higher using 180 days of data. On the other hand, LSTM had a higher decrease in terms of overall accuracy if compared to the use of the entire season dataset and the 180 or 120 days dataset. However, the accuracy remained the same for 180 and 120 days datasets.  Table 2. Overall Accuracy of LSTM and random forest(RF) classifiers using three different time windows as inputs.
In this context, some authors have already tested the random forest classifier for early season detection of different crops, achieving acceptable results (more than 85% overall accuracy). They found that for summer crops in Kansas, the USA, five months of data was the optimal time series for identifying crops, rather than the entire growing season (Hao et al., 2015).
In terms of accuracy per class, in the confusion matrices the recall values are plotted, demonstrating that for both classifiers the main problem was the attribution of less representative classes to the class 'Cultivated pasture' (Figures 2, 3). This result is expected since the cultivated pasture class has much more samples in the dataset, however, it was expected that the SMOTE technique was able to reduce this effect, as could be seen, in general, it was not.
Regarding the feature importance, there was no difference between time windows in terms of the selected band for optical data. For all the intervals, B11 (SWIR) had the higher importance (Table  3). The shortwave infrared region usually represents the content of water and other biochemical components present in the leaves and was already indicated as a good source of information for crop type classification (Zhang et al., 2017). Further, regarding the most important feature for each time interval, it could be seen that all three are at the beginning of the provided time series (October for the entire season and January for shorter time series which start in January). Regarding the ICLS areas, at this period, there are only annual crops planted, the pasture comes in March/April. Thus, this result could suggest that the Random Forest is not identifying the ICLS system but the annual crop instead. Since there are no other fields with a single crop than the ones with ICLS, the accuracy for ICLS is high.
Time Window Band Date Entire season B11 2019-10-25 180 days B11 2020-01-01 120 days B11 2020-01-01 Table 3. Entries with the highest feature importance in three different time windows. Finally, its important to highlight that this approach needs to be adapted accordingly to the growing season of each region, to assure that the temporal window comprises the vegetative peak of the annual crop.

Classifiers
The best overall accuracy for the RF classifier was obtained using 'nestimators'(The number of trees in the forest) equals 1000. For the LSTM net, the architecture was one LSTM layer (200), followed by one dropout layer (0.5), one dense layer (64), a batch normalization layer, another dense layer (128), another dropout layer (0.3), and finally a dense layer with the softmax activation function. Also, the batch size was 32 and 1000 epochs (early stopping applied). For the LSTM algorithm, more complex architectures were tested. However, it did not change the accuracy in more than 1% so, the simpler architecture was adopted (Zhong et al., 2019).
Concerning the performance difference of both classifiers, it could be seen that random forest achieved a better result than LSTM, if The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France we consider that they have similar Overall accuracy when RF is using only six months of data and LSTM the entire season dataset 2. Similar overall accuracy between random forest and LSTM for crop type mapping was already identified in other studies (Zhong et al., 2019). Also, it could be noticed that when reducing the length of the time windows, the accuracy of the LSTM net decreases more than the random forest Accuracy (Table 2). This result is expected since the LSTM net uses the time series to build its predictions (Sun et al., 2019). Thus, based on our results, it could be argued that maybe LSTM nets could be not so ideal for early season mapping.
In this perspective, there are three possibilities to explain the inferior performance of LSTM for the other classes and even for smaller time windows, (i) the architecture is too simple for this problem, (ii) the time series is too long for LSTM correctly detect dependencies among it, or, (iii) there is not sufficient data to train the model. Many authors highlighted that one of the main problems related to the use of Deep Learning is its high dependency on large amounts of data for train (Géron, 2019). However, it is interesting to notice that for our target class (ICLS) the LSTM net obtained a higher accuracy(0.98) than the RF classifier (0.87).

Prediction
Using the best model for each algorithm (Entire Season -LSTM, 180 days -Random forest), two predictions for the entire study area were made (Figure 4). It could be seen that LSTM net probably over predicted many shrub areas since, accordingly to ground reference data, there is more pasture areas than shrub pasture areas surrounding the study area (Figure 4 -A). However, it could be argued that maybe, LSTM net is distinguishing pasture areas with the presence or absence of cattle, identifying temporal variations in vegetation indices, but, more ground truth data is needed to validate this hypothesis. Also, for our main interest class (ICLS), if a comparison is made between the prediction from LSTM and random forest, the ones identified by RF are more gathered than the ones identified by LSTM, looks like both nets missed some ICLS areas, but RF was more coherent with the ground truth ( Figure 4). The prediction for the random forest algorithm closely represents the results in the confusion matrix (Figure 2), it means that the more frequent class in the map is Cultivated pasture. Probably some of those fields belong to a less representative class ( Figure  4). On the other hand, the prediction for the LSTM model clearly overpredicted shrub pasture fields, and this is not present in the confusion matrix ( Figure 3).
Also, it needs to be highlighted that some regions were not correctly identified, probably due to required corrections in the segmentation algorithm. This is the case of some wet areas/native forest fields (mostly at the top of Figure 4), where LSTM classified as ICLS and RF classified it as Eucalyptus or Cultivated pasture.

CONCLUSION
The aim of this study was to evaluate the potential of Sentinel-2 data to map integrated systems. The Random Forest and LSTM were able to identify ICLS with accuracy equal to or higher than 85%. Regarding the test of the early-season approach, we achieved high accuracy using only 180 days of data using the Random Forest classifier, half of which is commonly used by most published studies based on multi temporal classification, whose focus is on an entire season of data.
Comparing both classifiers, the Random Forest was superior to LSTM in the three tested periods (365 days: OA 86%; 180 days: The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLIII-B3-2022 XXIV ISPRS Congress (2022 edition), 6-11 June 2022, Nice, France OA 87%; 120 days: OA 86%). I.e., LSTM showed limitations for the early-season approach because of its architecture suggesting the test of more effective networks for our problem. Finally, based on our results, there are many different next steps to be suggested. Regarding the early-season approach, another study testing different time windows could be performed to achieve the highest accuracy as early as possible. Also, further studies could increase the complexity of the proposed LSTM net, intending to reach its maximum possible performance before discarding this net. Additionally, another neural net such as Transformer could be tested to identify ICLS, since this net is presenting promising results for crop type mapping recently.