Adaptive Resource Scheduling for Dual Connectivity in Heterogeneous IoT Cellular Networks

As massive distributed sensor devices are integrated into Internet for Internet of things (IoT) and generate tremendous data from simple measurement to rich multimedia information, wireless cellular networks like LTE are enforced to deploy more small cells to accommodate data from the countless IoT devices. In 3GPP Rel-12 specification, dual connectivity helps deploying the small cell eNBs by separating a control and data plane to a macro and small cell, respectively. The dual connectivity also improves per-user throughput and mobility robustness. Meanwhile, dynamic TDD configuration in the Rel-12 can enhance radio resource utilization of TDD-based small cells even though intercell interference can be worse than legacy static configuration within a small cell cluster. In this paper, we propose a heterogeneous cellular IoT network architecture using the aforementioned two small cell features, as well as scheduling algorithms for load balancing in the dual connectivity and for dynamic TDD configuration to mitigate interference in the small cell cluster. We evaluate proposed algorithms using LTE system level simulator and show that our approach improves network throughput.


Introduction
Megatrend of Internet of things (IoT) accelerates deployment of many sensors and actuators for various applications such as home and factory automation and unmanned vehicles and drones. Previously, wireless local area networking (WLAN) using Wi-Fi, Bluetooth, and Zigbee played an important role of connecting the IoT field devices to Internet. However, recently wireless wide area networking (WWAN) like GSM, 3G, and LTE receives much attention for outdoor sensors and robust and secure connections, which enables fast deployment of the field devices in wide area without efforts to establish the WLAN. For this reason, cellular IoT is now studied by 3GPP in a clean state for low data rate services such as metering in smart grid. Figure 1 illustrates wireless cellular IoT networks in urban area, where high-end IoT devices like vehicles, road-view camera, personal multimedia devices, and so forth and low-end devices like smart grid meters, road-sensors, and so forth coexist.
Exponentially increasing IoT devices and associate traffic will demand 10 times the wireless network capacity in 2017 compared to 2012 [1,2]. One of the most effective approaches to accommodate fast growing traffic from the massive devices is deploying small size cells (e.g., femto-, pico-, and microcells) [3]. In Figure 1, many small cells provide wide-band connections to local IoT devices within a single macro cell. 3GPP has studied small cell enhancement (SCE) about higher modulation scheme (e.g., 256 QAM), small cell on-off and discovery, radio interface based synchronization, and dual connectivity for release-12 (Rel-12) specification [4,5]. The dual connectivity to a Macro eNB (MeNB) and a Small eNB (SeNB) of a small cell in two-tier heterogeneous networks (HetNets) is useful to deploy the small cells autonomously for the IoT devices, in which the devices are controlled by the MeNB while data traffic is handled by the SeNB. Additionally, the dual connectivity improves per-user data throughput, mobility robustness, and signalling overhead reduction.
A TDD-based LTE system (i.e., TD-LTE) is efficient to utilize radio resources of uplink (UL) and downlink (DL) channels under varying traffic demands, which schedules flexibly transmission directions at each time slot (i.e, a subframe). The IoT devices for various applications can demand 2 International Journal of Distributed Sensor Networks temporarily asymmetric traffic rate in the UL and DL channels. The TD-LTE is appropriate for small cells operating in new bands 42 and 43 at 3400-3800 MHz that are TDD bands and FCC announced commissions rules with regard to commercial operations in the TDD bands on 3550-3650 MHz, Notice of Proposed Rulemaking and Order (NPRM), for small cell usage [6]. In order to support more dynamic UL-DL resource scheduling in small cells, 3GPP released a new standard about dynamic TDD configuration in the Rel-12, that is, enhancements to Interference Management and Traffic Adaptation (eIMTA), while a current LTE system only allows semistatic TDD configuration [7]. For the eIMTA, however, intercell interference in a small cell cluster can be considerable due to different transmission directions at each subframe.
In this paper, our contribution is developing three scheduling algorithms for the MeNB and SeNB in a two-tier HetNet to maximize network throughput of IoT device: (i) Load balancing mechanism in the dual connectivity: this mechanism explores feasible rate in the dual connections and provides IoT devices robust connectivity to Internet. Our algorithm improves network throughput of IoT devices more than a round-robin scheduling under long backhaul delay. (ii) Radio resource allocation based on UE traffic: the MeNB and SeNB assign radio Resource Blocks dynamically to varying traffic of IoT devices based on packet queue status and feedback about channel quality from the IoT devices.
(iii) Dynamic TDD configuration of TD-LTE SeNB: the dynamic TD-LTE system can exacerbate interference in a small cell cluster with autonomous small cells for own IoT devices due to different TDD configurations. Our algorithm reduces the interference between IoT device communications by arranging UL and DL transmissions while satisfying user demands.
The rest of this paper is organized as follows. Related works for this paper are introduced in Section 2. In Sections 3 and 4, architecture of a heterogeneous small cell network and TDD operation in LTE standard are described, respectively. We establish a system model and describe the aforementioned three algorithms in Section 5. Simulation results and analysis are shown in Section 6. We conclude in Section 7.

Related Works
Small cells including the indoor femtocells cover almost 60% of global traffic in 2015 [8]. Researches about small cell networks have been popularly performed as can be seen in many related publications and EU and US projects. 5G projects like METIS [2] also announced Ultra Dense Network (UDN) as a key feature of the next generation network, which is composed of many small cells with heterogeneous radio access technologies (RATs) [9].
For the dense small cell deployment, interference management is one of the major challenges to improve spectral efficiency and increase network capacity. In the cochannel deployment for macro and small cells in the HetNet, two types of interference exist: colayer (i.e., cotier) and cross-layer (i.e., cross-tier) interference. The colayer interference occurs between small cells, or a small cell and User Equipment (UE), while cross-layer interference is caused by both macro/small cells and UE [10][11][12][13]. In [14], a dense femtocell network with an overlaid macro cell performs worse than sparse femtocell network in terms of coverage and capacity due to the cochannel interference.
To reduce the cochannel interference in the HetNets, various approaches have been explored such as power control, frequency partition, and interference cancellation. There are many existing literatures about interference mitigation [12,[15][16][17][18]. References [16,19] propose a cochannel deployment algorithm of small cell networks controlling transmitting power based on distance to near macro cells. More power control algorithms for the HetNets have been proposed for both uplink [15,20] and downlink scenarios [21,22] based on optimization frameworks with objectives of minimizing the transmit power of small cells under a required SINR constraint. In addition, learning based noncooperative power control algorithms [23][24][25] were proposed. In 3GPP Rel-10 standard, enhanced intercell interference coordination (eICIC) technique has been introduced for interference mitigation in the HetNets. Many literatures investigate the eICIC technique, especially in time domain [18,[26][27][28].
In 3GPP Rel-12, study about small cell enhancement (SCE) in high layer introduced features of mobility enhancement, signalling overhead reduction, RRC diversity, and dual connectivity [29]. The dual connectivity allows UE to have dual connections to a MeNB and SeNB. Using the dual connections, the UE can achieve higher data rate than a single connection and separate a control and data path. From this, control messages are delivered to the MeNB while user data are served by the SeNB, which is called user plane (UP)/control plane (CP) split. Accordingly, UE can be more robust in mobility between small cells or small and macro cells because an overlaid MeNB manages UE mobility (i.e., location and radio resources) as an anchor point, which prevents frequent handover procedures between small cells [30]. Furthermore, signalling overhead to core networks caused by frequent handovers can be mitigated as the MeNB controls mobility on behalf of a serving gateway (SGW). For uplink transmission, dual connectivity is not favourable due to higher power consumption compared to limited UE battery. Uplink transmission only to the small cells consumes less power because UE is closer to the small cells than to the macro cell.
In small cell deployment, small cells can be branched from a macro cell or directly from a core network (CN), that is, SGW, which are called "radio access network (RAN) split" and "CN split," respectively. A backhaul in the CN split can carry larger user traffic than the RAN split. However the CN split increases control signal overhead to deal with frequent UE mobility among small cells. In contrast, the RAN split can experience bottleneck in the MeNB with many attached small cells, but control overhead is less compared to the CN split. Backhaul latency between the MeNB and SeNB or SeNBs limits cooperation between eNBs such as collaborative scheduling for radio resources and joint transmission/reception [4,8].
Since 3GPP specified the eIMTA feature for TDD-based small cells [7], many researches have been conducted. Reference [31] investigates performance of dynamic TDD and higher modulation in homogeneous small cell networks. Simulation results show that small cell clustering can reduce intercell interference efficiently with low cost and interference cancellation also performs well in higher modulation schemes. For heterogeneous small cell networks, authors in [32] measure colayer/cross-layer interference between dynamic TDD-based small cells and between macro and small cells and propose an algorithm of the TDD configuration to reduce the interference based on the measurement. References [33,34] propose a close-loop power control scheme for UL transmission to reduce mutual interference among dynamic TDD-based small cells. In [34], authors also proposed adaptive transmission power control on subframes, which allows the eNBs and UE to adjust the transmission power differently to each subframe according to TDD configuration. Simulation results show that the adaptive power control in each subframe is more effective than a legacy fixed approach. Threshold based interference management is proposed [35], where a legacy overload indication is extended for each subframe to notify degree of interference at each dynamic TDD subframe. Previous researches about the dynamic TD-LTE mostly focus on transmission power control based on interference notification at subframe level. In this study, we investigate UE traffic steering methods to maximize throughput in HetNets where link capacity is varying due to the dynamic TDD operation.

Heterogeneous Small Cell Networks
Wireless heterogeneous networks (HetNets) consist of macro and underlying small cells (e.g., micro-, pico-, and femtocells) as shown in Figure 2 [36]. The Small eNBs (SeNBs) form dense cell clusters at hot spots with minimum 20 m intersite distance (ISD) within the macro coverage. The Macro eNB (MeNB) uses a Frequency Division Duplexing (FDD) approach, for instance, with 1.9 GHz and 2.1 GHz for UL and DL carrier frequencies, while the SeNBs operate at 3.5 GHz TDD spectrum (i.e., bands 42 and 43). In the figure, carrier frequencies are denoted, respectively, by 1 and 2 for the macro and small cells. Herein UEs are distributed over an entire network, but many of them are located at the hot spots, which allows them to be served simultaneously by the MeNB and SeNB.
In 3GPP Rel-10 standard for LTE-Advance (LTE-A), the Carrier Aggregation (CA) technology was introduced to provide a wider bandwidth than legacy LTE Rel-8 standard, which aggregates several small consecutive or nonconsecutive LTE bands to increase peak throughput [36,37]. In practice, the CA is realized by transceiving simultaneously multiple streams through multiple serving cells (i.e., maximum 5 cells) (number of aggregation carriers is considered up to 32 in 3GPP Rel-13.) that operate at different bands. For this, MAC layer splits a single bearer into different physical carriers for concurrent transmission.
In contrast to the CA where a single eNB manages the multiple serving cells, serving cells belong to different eNBs in the dual connectivity; the MeNB manages macro cells and the SeNB does small cells. The MeNB controls multiple underlying SeNBs for adding or releasing serving cells of SeNBs for UE using a backhaul connection 2 between the eNBs [38]. In a detailed procedure of the dual connectivity establishment, UE first reports measurement results about serving small cells of the SeNBs to the MeNB. If an access channel to a new small cell of the SeNB is qualified, the MeNB commands the UE to open a new connection to the small cell while making the SeNB assign radio resources for the connection. The MeNB distributes UE traffic among the dual connections adaptively based on traffic load and wireless channel quality. Instead of the MeNB, a serving gateway (SGW) can split the UE traffic flow for dual connectivity if the SeNB is connected to the SGW directly according to network deployment, that is, CN split.
The CA is more efficient to use multicarriers compared to dual connectivity because resource scheduling for the multicarriers is governed by a single eNB and transmission delay in the carriers is almost the same. Thus, CA can schedule packet forwarding among the multicarriers optimally based on channel quality. On the contrary, the split bearer in the dual connectivity can suffer different transmission delay and different scheduling at the MeNB and SeNB due to backhaul delay, more or less 20 msec. However, the dual connectivity is attractive for small cell deployment using stand-alone SeNBs that can be autonomously installed compared to the CA that requires fiber optic cabling between MeNB and a remote radio header (RRH) for a small cell.

Time Division LTE for Small Cells
As a well-known advantage of the TD-LTE (i.e., Time Division Duplex-(TDD-) based LTE), eNB can schedule dynamically UL and DL radio resources according to required bandwidth for UE's traffic demands [39,40]. FDD-based LTE has separate UL and DL carriers, in which the UL can be idle while the DL carrier is congested, or vice versa. Load asymmetry in UL and DL can be varying with applications, mobility, and time. Figure 3 shows a radio frame format for the TD-LTE. The radio frame consists of 10 subframes and each subframe is a base schedule unit (i.e., transmission time interval (TTI) = 1 msec) in time domain. The subframe has 14 orthogonal frequency division   In the TD-LTE, subframes can be configured for different transmission directions. Thus, the TD-LTE needs a special subframe like subframe 1 in Figure 3 to change transmission directions from DL to UL, which provides a time gap for RF path switching and uplink timing acquisition. In detail, the special subframe is composed of dwPTS (downlink pilot time slot), gap, and upPTS (uplink pilot time slot) where the dwPTS and upPTS are used for sending DL synchronization signals and UL sounding signals, respectively. However, transmission direction switch from UL to DL does not require an additional subframe.

TD-LTE TDD Configuration.
The TD-LTE defines 7 different UL and DL configurations [39] which have different ratios of UL and DL subframes within a single radio frame. Table 1 shows the TDD configuration, where 10 columns indicate a transmission direction, that is, "D" (DL) or "U" (UL) at each subframe in a single radio frame and the UL and DL ratio is shown in the parentheses. In addition, the special subframe is denoted as "S" upon changing the direction from DL to UL.
TD-LTE throughput depends on the TDD configuration and round trip time (RTT) at MAC layer. LTE system needs at least 4 msec for a single data transmission: 1 msec for sending data and 3 msec for processing received data. Thus, the RTT normally takes 8 msec for sending data and receiving an acknowledgement in FDD-based LTE system if retransmission does not occur. In the TD-LTE, however, the RTT is varying according to the TDD configuration. For example, the longest RTT can be more than 17 msec in TDD Configuration 5, which has only one subframe for an uplink transmission as shown in Table 1. Supposing that downlink data arrive at the last subframe 9, UE cannot send the acknowledgement in next radio frame because the UE needs 3 subframes for data processing and completes the processing  after uplink subframe 2. Thus, the UE has to wait one more radio frame. As a consequence, Configuration 5 shows worse downlink data rate than Configurations 2 and 3 [41]. But Configuration 5 performs similarly to Configuration 0 that has less downlink subframes than Configurations 2 and 3. A current LTE system configures the TDD configuration semistatically, in which the configuration does not change during operation. Recently, 3GPP released Rel-12 specification of dynamic TDD configuration, that is, enhancements to Interference Management and Traffic Adaptation (eIMTA) to improve resource utilization in UL/DL channels [7]. This standard allows eNBs and UE to configure UL and DL subframes dynamically based on user traffic demands. This dynamic TDD configuration can introduce intercell interference among neighbour TDD-based small cells compared to the previous semistatic configuration. In Figure 4, for example, two small cells with different transmission directions due to dynamic TDD configuration cause mutual interference. Small cell 1, 1, receives interference from DL transmission from small cell 2, 2, while UE 2, 2, also experience interference due to UE 1, 1, UL transmission at the same time. Therefore, it is necessary to orchestrate the TDD configuration among neighbour small cells to reduce the intercell interference.

System Model
In this section, we develop a system model of dual connectivity in heterogeneous wireless small cell networks, in which macro and small cells adopt different duplex schemes, FDD and TDD, respectively, and operate at different carrier frequencies. Thus, cross-layer interference between a macro and a small cell is not present while colayer interference between the MeNBs or SeNBs still exists. In the HetNet, the colayer interference among the small cells can be considerable since they form a cluster with a short ISD in urban environment [42].
We propose solutions for three problems in our system model to maximize throughput of dual connectivity in the two-tier HetNet: (i) a bearer split algorithm in a gateway for load balance in the dual connections, (ii) a radio resource scheduling algorithm at the MeNB and the SeNB based on backlogs and channel feedback, and (iii) a TDD configuration algorithm of a SeNB to mitigate mutual interference among small cells. First, downlink data of UE can be delivered by macro and small cells if the UE is located on an overlapped area of both cells and has dual connections. A MeNB or SGW distributes the downlink flow in the dual connections using a back-pressure algorithm that schedules data forwarding based on backlogs of queues at each eNB [43][44][45]. Second, the separate flows of the UE at the MeNB and the SeNB are scheduled by channel status given by a feedback from the UE such as Channel Quality Indication (CQI). The CQI is reported by eNB request or periodically. Third, the SeNB capacity for the outgoing flows is affected by TDD configuration of adjacent SeNBs in a small cell cluster. We assume that the SeNBs do not have any interface to each other to arrange their TDD configuration since many SeNBs can be deployed autonomously [42]. Instead, the MeNB that has 2 interfaces to the SeNBs can control their TDD configuration in a centralized approach. We propose a heuristic algorithm performed at the MeNB, which is based on branch and bound scheme to search an asymptotically optimal set of the TDD configuration efficiently. Table 2 shows variable definition of our system model. Serving MeNB and SeNB are indexed by and that belong to a set of eNBs, and , respectively. TDD configuration set is = {0, 1, 2, 3, 4, 5, 6} and a configuration of each SeNB is denoted as . A subframe of the SeNB has uplink or downlink transmission according to the TDD configuration .

Dual-Path Forwarding
Algorithm. First, a well-known back-pressure algorithm is adopted for bearer split in the dual connectivity. In our system model, a gateway who schedules packet forwarding in the dual connections can be an independent network equipment or embedded in the MeNB. The gateway forwards packets to UE via the links ( , ) of dual connections based on difference of queue levels and achievable flow rate as shown in Figure 5. Equation (1) maximizes throughput of multicommodity flows in the dual  connections, where ( , ) is an achievable flow rate in the link ( , ) and Ω is a convex set that satisfies constraints (2). The achievable flow rate is limited by the link capacity ( , ) , where the link ( , ) is a virtual link rather than a physical backhaul link and its capacity is determined by feasible wireless link capacity for the flow . The virtual link is located on the MeNB internally or on a physical Ethernet link between the gateway and MeNB or between the gateway and SeNB: (2) Algorithm 1 shows a packet forwarding algorithm of the gateway for dual connectivity. The gateway receives control packets to calculate differential backlogs from the MeNB and the SeNBs. Then, the gateway chooses a path based on the differential backlog and flow rate on each link to eNB. In practice, the gateway can use average flow rate based on transmission rate of the virtual link given by eNBs.

Radio Resource Scheduling
Algorithm. Second, we propose maximum weight based PRB scheduling for each eNB, which decides allocation of PRB , to maximize link capacity for the flow . The following model presents a scheduling algorithm at the TDD-based SeNB, but the MeNB also follows the same approach: ∀ ∈ , = ( ) , ∑ ∈ ( ) ≤ 1, ∀ ∈ , ∀ ∈ .
In (3), the objective is maximizing a sum of multiplication of a queueing status and access link capacity for each flow. The access link capacity is sum of the assigned PRBs per flow in (4). ( ) is a binary variable presenting the PRB allocation: 1 if PBR is allocated to UE flow and 0 otherwise. The association vector is 1 if a flow is connected to eNB, , and 0 otherwise. is UE of the flow .
Each PRB capacity , ( ) at the eNB is derived approximately by Shannon equation of (8). Capacity in a real LTE system is less than this theoretical value, for example, about 7 dB: where is a PRB, ( ) is its bandwidth, and Signal to Interference and Noise Ratio (SINR) is received signal quality at the PRB in the UE . In constraint (6), each UE flow has a single access to a SeNB at an allocated PRB. In practice, UE associates with a single SeNB in the dual connectivity based on Reference Signal Received Power (RSRP) and Reference Signal Received Quality (RSRQ). The next constraint (7) assumes unicast for UE flow. In other words, a PRB within a SeNB is allocated only to a single UE flow. In LTE networks, most of UE traffic is still handled by unicast rather than multicast.
This second problem for PRB allocation can be solved by a mixed integer programming, which is NP-hard problem.
Algorithm 1: Scheduling algorithm for bearer splitting.
COMMENT: Decide number of scheduling radio blocks (SRB) Thus, we develop a simple iterative algorithm for the PRB allocation as shown in Algorithm 2. eNB calculates a weight value of each UE flow with backlogs and access link capacity of the UE flow. Then, the algorithm chooses maximum weight among the flows and assigns the Resource Block. After then, the algorithm updates queue state of the flow. The UE access link capacity is estimated by the CQI feedback, periodically reported by the UE, for example, 2, 5, 10, 20, 40, and 80 msec. Since the CQI is reported per each subband or full band, radio resources need to be scheduled by the subband that includes different number of PRBs according to system bandwidth (e.g., 4 PRBs/5 MHz or 6 PRBs/10 MHz). Therefore, the MeNB and SeNB should decide PRB allocation with the number of subbands and CQI period. For example, suppose that a SeNB with 25 PRBs has approximately 6 scheduling Resource Blocks (SRBs) every subframe and the CQI period is 2 msec; a total of 12 scheduling blocks should be determined by Algorithm 2. This algorithm provides a weight fair scheduling with an average UE flow rate, = / ∑ ̸ = ⋅ ( ), even if UE at cell edge can achieve limited throughput due to low SINR.

Dynamic TDD Configuration
Algorithm. Third, we propose a centralized algorithm of SeNB TDD configuration at a MeNB to maximize sum of access link capacity as shown in (9). In (10), the access link capacity for each flow is sum of link capacity at each subframe during configuration period, . Since PRB allocation for each link is done by Algorithm 2, the link capacity is determined only by SINR that is varying with surrounding SeNBs or UEs that generate interference as can be seen in (11). Thus, our algorithm finds out an optimal which can improve link capacity of the small cells.
The binary variables , and , indicate a transmission direction with configuration at a subframe ; for example, , is 1 if the subframe is an uplink subframe and 0 otherwise. In (13), there is interference from an adjacent SeNB if , is 1 and from an adjacent UE if , is 1: , = ∑ , ( ) , ∈ {0, 1, 2, . . . , } , , , ∈ {0, 1} , ∀ ∈ , ∀ ∈ (12) The DL SINR of a PRB can be different by TDD configuration of adjacent small cells as shown in (13), where , is DL transmitting power at the SeNB and is a path loss and gain. And noise power is added. Downlink interference from neighbour SeNBs and UE is varying with . UL SINR can be similarly derived.
This TDD configuration problem can be solved by mixed integer nonlinear programming (MINLP), which complexity increases exponentially as the number of SeNBs increases. We propose a heuristic algorithm based on a best first search with branch and bound, which reduces scheduling complexity in practice by reducing number of cases of SeNB TDD configuration. Algorithm 3 describes details of the centralized algorithm running at the MeNB or the SGW. For this, SeNBs should report UE PRB allocation and measurement information periodically to a scheduler, the MeNB or the SGW, through 2 interface like in inter-eNB Coordinated Multipoint (CoMP) operation. First, the algorithm selects a SeNB cluster based on given UE flow rate and UE channel quality, which has highest aggregated flow rate and most adjacent SeNBs, and decides a root TDD configuration of all SeNBs as a start point. That is to say, a dense small cell area with more UE is scheduled first rather than sparse area. In the search space of TDD configuration of SeNBs, sum of achievable UE flow rates is derived as a bound value . With the initial configuration and the bound value, the algorithm visits adjacent SeNBs one by one in the cluster and decides a TDD configuration according to the given UE channel quality and traffic demands. The algorithm cuts evaluating individual TDD configuration of a SeNB that is already visited and a flow-sum value in current configuration is equal to or less than bound value as pruning for search reduction. Once SeNB in the cluster is configured, the algorithm finds next TDD configuration node which includes unvisited SeNBs in next cluster. The algorithm is completed when all SeNBs are visited. This algorithm can be performed every several ten milliseconds due to backhaul delay between the MeNB and SeNB. Accordingly, faster scheduling can be achieved using fiber optical cables like in the intra-eNB CoMP.

Simulation and Results
We developed LTE system level simulator of two-tier HetNets and implemented our three scheduling algorithms in the MeNB and SeNB: Algorithms 1 and 2 for the MeNB and Algorithms 2 and 3 for the SeNB. Detailed parameters for our LTE HetNet simulator are shown in Table 3 including a channel model [5]. We placed 15 SeNBs and 120 active UEs randomly within a sector of the MeNB, where UEs have dual connectivity depending on their location. Considering the MeNB cell radius, average SeNB ISD can be more than 50 m according to distribution.
The MeNBs and SeNBs have 25 and 50 Physical Resource Blocks (PRBs) with 5 and 10 MHz bandwidth, respectively. The MeNB has separate UL and DL channels with the same bandwidth for FDD operation while the TDD-based SeNB has to share the 10 MHz bandwidth for UL and DL channels.
For the Hybrid Automatic Repeat Request (HARQ) operation in TDD-based small cells, format 3 for Uplink Control Information (UCI) is assumed to be used for simplicity in this simulation because acknowledgement bundling in the TDD system may require massive bits for the feedback at worst case. Using 50 RBs with format 3, only 50 UEs can acknowledge received data in a single uplink subframe. In contrast, it will be 300 UEs (50 RBs × 6 multiplex factors per RB) if format 1b with channel selection is used. With format 1b, shortage of UL feedback channels could not occur if there is no much UL data traffic.
We generate DL and UL traffic with source rate 1 Mbps and 500 Kbps for each UE based on Poisson distribution, where each packet size is set by 500 bytes as a datagram size.   Figure 6 shows distribution of the UE and the SeNBs in the network for simulation. Figures 6(a) and 6(b) show CDF of reachable SeNBs from UE and CDF of the number of UEs attached to each SeNB, respectively. The reachable SeNBs are candidate eNBs for UE to connect, which are reachable at both UL and DL. The UE associates with one of them based on received signal strength. Thus, transmissions by the UE or the reachable SeNBs can interfere with each other. According to Figure 6(a), most of UEs have from 1 to 5 SeNBs to access and around 40% of UEs do not have SeNBs nearby. In Figure 6(b) each SeNB has different number of associated UEs. Almost a half of total SeNBs have two associated UEs and 10% SeNBs have more than 5 UEs. Figure 7(a) shows average received data with different forwarding scheduling methods for dual connectivity like "SeNB only," "round-robin," and "back-pressure." In this figure, we depict that the user data rate is converged into the saturated value by our algorithms during the 100 subframes. In the SeNB only, the MeNB sends packets to only a SeNB. And the MeNB sends a packet alternatively to the SeNB or the MeNB in the round-robin. The back-pressure algorithm as our proposal achieves around 20% better performance than others. The round-robin scheme performs worse than even the SeNB only since cell capacity of the SeNB that is larger than the MeNB is not utilized sufficiently. Only one or two UEs can use entire SeNB resources as observed in Figure 6(b). Figure 7(b) shows backlog status in the MeNB and the SeNB with two different packet forwarding algorithms, round-robin and back-pressure. The round-robin approach performs worse than the back-pressure in terms of the queued data in the eNBs, which are fast growing at the congested MeNB. But the back-pressure approach shows consistent queueing in both eNBs. Backlogs in the MeNB can degrade eventually TCP performance by increasing RTT and retransmission timeout (RTO). Figure 8 illustrates average received data in the UE with different radio resource allocation mechanisms. One is a simple "round-robin" that assigns PRBs of each subframe to attached UEs almost evenly. The other is "weight scheduling" as our proposed approach in Algorithm 2, which allows each eNB to select UE who has larger backlog and better channel status (e.g., high modulation order) to maximize throughput. In this simulation, we experiment a scenario with 5 Mbps source rate for more workload in addition to the 1 Mbps. As shown in Figure 8(    received data from the MeNB and SeNB under higher traffic load, 5 Mbps. The weight scheduling could achieve more gains from spatial diversity in the MeNB compared to the SeNB, which gains more than 50% compared to the round-robin while it gets more or less 35% in the SeNB. Figure 9 shows simulation results from Algorithm 3 for SeNB TDD configuration scheduling. Figure 9(a) shows that our proposed algorithm improves network throughput as TDD configuration decided by our algorithm is applied dynamically to SeNBs every radio frame (10 ms) that is a minimum period of dynamic configuration defined in 3GPP eIMTA specification. In the simulation topology, our algorithm could exploit search space of TDD configuration for 15 small cells within a second although several overlapped International Journal of Distributed Sensor Networks SeNBs cause revisit. As can be seen in the figure, the aggregated received data of UEs increase progressively as TDD configuration of each SeNB is decided. At the end of scheduling, the received data per frame increase by around 15% compared to initial throughput even if the number of adjacent SeNBs is not so high as shown in Figure 6. To compare metaheuristic and our algorithm, Figure 9(b) shows average received data with two different scheduling methods, "random" and "dynamic configuration" while two different splitting algorithms in dual connectivity are applied, "roundrobin (RR)" and "back-pressure (BP)." The back-pressure scheduling for dual connectivity performs better than the round-robin by about 30% in dynamic TDD configuration. The average received data in dynamic configuration and backpressure is varying in time compared to the other since the TDD configuration is affected by the PRB allocation status that is varying with the queued data.

Conclusion
In this study, we proposed an architecture of heterogeneous IoT cellular networks. To maximize user throughput and provide robust connectivity for IoT devices, we developed a packet forwarding algorithm for dual connectivity and scheduling algorithms for radio resources and TDD configuration of TD-LTE small cells in the heterogeneous networks. Using LTE system level simulation, we demonstrated feasibility of our proposed algorithms with results of improved per-user and network throughput. For future works, we will investigate stability of our algorithms in varying user traffic demand and mobility. Also, we will compare our algorithms with optimal solutions in terms of performance and complexity in time and memory.